I am working on the file "Notebooks/Viromics/analysis_CirSeqAndCloneSamples.ipynb".
The first task is:
Write out the table
Can you write Python code for it?

def write_increasing_positions(increasing_pos, tables, exp_names, filename) :
    increasing_pos_table = pd.DataFrame()

    for d in tables:
        increasing_pos_table = pd.concat([increasing_pos_table,d[d['Position'].isin(increasing_pos)]])

    num = len(increasing_pos)

    expnames_iterated = []
    for i in exp_names :
        expnames_iterated.append([i]*num)

    expnames = [item for sublist in expnames_iterated for item in sublist]
    print(len(expnames))
    increasing_pos_table['expName']=expnames
    increasing_pos_table.to_csv( filename )

write_increasing_positions(increasing, tables_clone, all_table_names, "increasing_clone.csv")

printMajorFrequencyThroughSamples_2340_7172(tables_clone)


I am working on the file "Notebooks/Viscous Nutation Damper.ipynb".
The first task is:
Angular momentum of the wheel:
Can you write Python code for it?


I am working on the file "Notebooks/Viscous Nutation Damper.ipynb".
The first task is:
Calculate total angular momentum:
Can you write Python code for it?

hG_sat = Isat_B*iWb + Iwheel_B*bWa
hG_sat


I am working on the file "Notebooks/Viscous Nutation Damper.ipynb".
The first task is:
Define ${}^\mathcal{I}\boldsymbol{\omega}^\mathcal{B}$ and ${}^\mathcal{B}\boldsymbol{\omega}^\mathcal{A}$ where $\mathcal A$ is the frame fixed to the wheel ($\mathbf b_3 \equiv \mathbf a_3$):
Can you write Python code for it?

iWb = Matrix([w1,w2,w3])
bWa = Matrix([0,0,O])
iWb,bWa


I am working on the file "Notebooks/Viscous Nutation Damper.ipynb".
The first task is:
Define total and wheel MOI in $\mathcal B$ frame (about respective centers of mass):
Can you write Python code for it?

Isat_B = diag(It,Is,It)
Iwheel_B = diag(I1,I1,Iw)
Isat_B,Iwheel_B


I am working on the file "Notebooks/Viscous Nutation Damper.ipynb".
The first task is:
Finally, some numerical integration:
Can you write Python code for it?

import numpy, scipy
import scipy.integrate
import matplotlib
%matplotlib inline
import matplotlib.pyplot as plt


I am working on the file "Notebooks/Viscous Nutation Damper.ipynb".
The first task is:
Pick some numerical values and initial conditions:
Can you write Python code for it?

IsItn = 1.5
#IsItn = 1/1.5
IwItn = 0.06
IwIsn = IwItn/IsItn
DItn = 0.5

z0n = [0.2, 2, 0, 0]; #rad/s

dz1 = dz.subs(([IsIt,IsItn],[IwIt,IwItn],[IwIs,IwIsn],[DIt,DItn]))
dz1

f1 = lambdify((w1,w2,w3,O),dz1)
f2 = lambda z,t: f1(z[0],z[1],z[2],z[3]).flatten()

t = numpy.linspace(0,400,1000)
y = scipy.integrate.odeint(f2,z0n,t)

plt.plot(t, y[:, 1], 'b', label='$\\omega_2(t)$')
plt.legend(loc='best')
plt.xlabel('Time (s)')
plt.ylabel('radians')
plt.grid()

plt.plot(t, y[:, 0], label='$\\omega_1(t)$')
plt.plot(t, y[:, 2], label='$\\omega_3(t)$')
plt.legend(loc='best')
plt.xlabel('Time (s)')
plt.ylabel('radians')
plt.grid()

plt.plot(t, y[:, 3], label='$\\Omega(t)$')
plt.legend(loc='best')
plt.xlabel('Time (s)')
plt.ylabel('radians')
plt.grid()


I am working on the file "Notebooks/Visualization/Data Visualization Worksheet - Answers.ipynb".
The first task is:
Adding Subplots
The first thing you'll need to do is call the .subplots( nrows=<rows>, ncols=<cols> ) to create a subplot.
Next, plot your charts using the .plot() method.  To do add the second plot to your figure, add the ax=axes[n] to the .plot() method.
Can you write Python code for it?

fig, axes = plt.subplots(nrows=1, ncols=2)

finalData['ConfickerAB'].plot()
finalData['Bedep'].plot(ax=axes[0])

from bokeh.plotting import output_notebook
output_notebook()

from bokeh.charts import TimeSeries
from bokeh.io import show

linechart = TimeSeries( data=finalData,
                       title="ConfickerAB Hosts",
                       legend="top_left" )
show( linechart )


I am working on the file "Notebooks/Visualization/Data Visualization Worksheet - Answers.ipynb".
The first task is:
Create the first chart
Using the .plot() method, plot your dataframe and see what you get.
Can you write Python code for it?


I am working on the file "Notebooks/Visualization/Data Visualization Worksheet - Answers.ipynb".
The first task is:
Data Visualization Worksheet
This worksheet will walk you through the basic process of preparing a visualization using Python/Pandas/Matplotlib.
For this exercise, we will be creating a line plot comparing the number of hosts infected by the Bedep and ConfickerAB Bot Families in the Government/Politic sector.
Prepare the data
The data we will be using is in the dailybots.csv file which can be found in the data folder.  As is common, we will have to do some data wrangling to get it into a format which we can use to visualize this data.  To do that, we'll need to:
1.  Read in the data
2.  Filter the data by industry and botnet
The result should look something like this:
| |date|ConflikerAB|Bedep|
|-|----|-----------|-----|
|0|2016-06-01|255|430|
|1|2016-06-02|431|453|
The way I chose to do this might be a little more complex, but I wanted you to see all the steps involved.
Step 1 Read in the data
Using the pd.read_csv() function, you can read in the data.
Can you write Python code for it?

data = pd.read_csv('../../data/dailybots.csv')
data.head()


I am working on the file "Notebooks/Visualization/Data Visualization Worksheet - Answers.ipynb".
The first task is:
Move the Legend to the Upper Center of the Graph
For this, you'll have to assign the plot variable to a new variable and then call the formatting methods on it.
Can you write Python code for it?

nicePlot = finalData.plot( kind="line")
nicePlot.legend(loc='upper center', bbox_to_anchor=(0.5, 1.05),
          ncol=3, fancybox=True, shadow=False)


I am working on the file "Notebooks/Visualization/Data Visualization Worksheet - Answers.ipynb".
The first task is:
Next, I created a second DataFrame which only contains the information from the ConfickerAB botnet.  I also reduced the columns to the date and host count.  You'll need to rename the host count so that you can merge the other data set later.
Can you write Python code for it?

filteredData2 = filteredData[filteredData['botfam']== 'ConfickerAB' ][['date','hosts']]
filteredData2.columns = ['date', 'ConfickerAB']
filteredData2.date = pd.to_datetime( filteredData2.date )
filteredData2.head()


I am working on the file "Notebooks/Visualization/Data Visualization Worksheet - Answers.ipynb".
The first task is:
Repeat this porcess for the Bedep botfam in a separate dataFrame.
Step 3: Merge the DataFrames.
Next, you'll need to merge the dataframes so that you end up with a dataframe with three columns: the date, the ConfickerAB count, and the the Bedep count.  Pandas has a .merge() function which is documented here: http://pandas.pydata.org/pandas-docs/stable/merging.html
Can you write Python code for it?

filteredData3 = filteredData[filteredData['botfam']== 'Bedep' ][['date','hosts']]
filteredData3.columns = ['date', 'Bedep']
filteredData3.date = pd.to_datetime( filteredData3.date )
finalData = pd.merge(filteredData2, filteredData3, on='date', how='left')
finalData.head()


I am working on the file "Notebooks/Visualization/Data Visualization Worksheet - Answers.ipynb".
The first task is:
Step 2:  Filter the Data
The next step is to filter both by industry and by botfam.  In order to get the data into the format I wanted, I did this separately.  First, I created a second dataframe called filteredData which only contains the information from the Government/Politics industry.
Can you write Python code for it?

filteredData = data[data['industry'] == "Government/Politics"]
filteredData.head()


I am working on the file "Notebooks/Visualization/Data Visualization Worksheet - Answers.ipynb".
The first task is:
Step 3 Customizing your plot:
The default plot doesn't look horrible, but there are certainly some improvements which can be made.  Try the following:
1.  Change the x-axis to a date by converting the date column to a date object.
2.  Move the Legend to the upper center of the graph
3.  Make the figure size larger.
4.  Instead of rendering both lines on one graph, split them up into two plots
5.  Add axis labels
There are many examples in the documentation which is available: http://pandas.pydata.org/pandas-docs/version/0.18.1/visualization.html
Can you write Python code for it?

finalData.set_index('date', inplace=True)
finalData.plot(kind='line' )


I am working on the file "Notebooks/Word_Prediction_Add-1_Smoothing_with_Interpolation.ipynb".
The first task is:
For Debugging Purpose Only
Ignore running the cells below if not debugging
Can you write Python code for it?

#variable declaration
vocab_dict = defaultdict(int)          #for storing the different words with their frequencies
bi_dict = defaultdict(int)             #for keeping count of sentences of two words
tri_dict = defaultdict(int)            #for keeping count of sentences of three words
quad_dict = defaultdict(int)           #for keeping count of sentences of four words
quad_prob_dict = defaultdict(list)         #for storing the probable  words for Quadgram sentences
tri_prob_dict = defaultdict(list)          #for storing the probable  words for Trigram sentences
bi_prob_dict = defaultdict(list)           #for storing the probable  words for Bigram sentences


I am working on the file "Notebooks/Word_Prediction_Add-1_Smoothing_with_Interpolation.ipynb".
The first task is:
<u>Do preprocessing</u>:
Remove the punctuations and lowercase the tokens
Can you write Python code for it?

#returns: string
#arg: string
#remove punctuations, change to lowercase ,retain the apostrophe mark
def removePunctuations(sen):
    #split the string into word tokens
    temp_l = sen.split()
    #print(temp_l)
    i = 0
    j = 0

    #changes the word to lowercase and removes punctuations from it
    for word in temp_l :
        j = 0
        #print(len(word))
        for l in word :
            if l in string.punctuation:
                if l == "'":
                    if j+1<len(word) and word[j+1] == 's':
                        j = j + 1
                        continue
                word = word.replace(l," ")
                #print(j,word[j])
            j += 1

        temp_l[i] = word.lower()
        i=i+1

    #spliting is being done here beacause in sentences line here---so after punctuation removal it should
    #become "here so"
    content = " ".join(temp_l)

    return content


I am working on the file "Notebooks/Word_Prediction_Add-1_Smoothing_with_Interpolation.ipynb".
The first task is:
<u>Driver Function for Testing the Language Model</u>
Can you write Python code for it?

#return: void
#arg:string,string,dict,dict,dict,dict,dict
#Used for testing the Language Model
def trainCorpus(train_file,test_file,bi_dict,tri_dict,quad_dict,vocab_dict,prob_dict):
    score = 0

    #load the training corpus for the dataset
    token_len = loadCorpus(train_file, bi_dict, tri_dict, quad_dict, vocab_dict)
    print("---Processing Time for Corpus Loading: %s seconds ---" % (time.time() - start_time))

    start_time1 = time.time()

    #estimate the lambdas for interpolation
    #found earlier usign estimate Function
    param = [0.7,0.1,0.1,0.1]
    #param = estimateParameters(token_len, vocab_dict, bi_dict, tri_dict, quad_dict)
    #print(param)

    #create trigram Probability Dictionary
    findTrigramProbAdd1(vocab_dict, bi_dict, tri_dict, tri_prob_dict)
    #create bigram Probability Dictionary
    findBigramProbAdd1(vocab_dict, bi_dict, bi_prob_dict)
    #create quadgram Probability Dictionary
    findQuadgramProbAdd1(vocab_dict, bi_dict, tri_dict, quad_dict, quad_prob_dict)
    #sort the probability dictionaries
    sortProbWordDict(bi_prob_dict, tri_prob_dict, quad_prob_dict)
    gc.collect()
    print("---Preprocessing Time for Creating Probable Word Dict: %s seconds ---" % (time.time() - start_time1))

    ### TESTING WITH TEST CORPUS
    test_data = ''
    #Now load the test corpus
    with open('test_corpus.txt','r') as file :
        test_data = file.read()

    #remove punctuations from the test data
    test_data = removePunctuations(test_data)
    test_token = test_data.split()

    #split the test data into 4 words list
    test_token = test_data.split()
    test_quadgrams = list(ngrams(test_token,4))

    #choose most probable words for prediction
    start_time2 = time.time()
    score = computeTestScore(test_quadgrams,bi_dict, tri_dict, quad_dict, vocab_dict,
                     bi_prob_dict, tri_prob_dict, quad_prob_dict, token_len,param)
    print('Score:',score)
    print("---Processing Time for computing score: %s seconds ---" % (time.time() -...


I am working on the file "Notebooks/Word_Prediction_Add-1_Smoothing_with_Interpolation.ipynb".
The first task is:
<u>For Taking input from the User</u>
Can you write Python code for it?

#for taking input from user

#returns: string
#arg: void
def takeInput():
    cond = False
    #take input
    while(cond == False):
        sen = input('Enter the string\n')
        sen = removePunctuations(sen)
        temp = sen.split()
        if len(temp) < 3:
            print("Please enter atleast 3 words !")
        else:
            cond = True
            temp = temp[-3:]
    sen = " ".join(temp)
    return sen


I am working on the file "Notebooks/Word_Prediction_Add-1_Smoothing_with_Interpolation.ipynb".
The first task is:
<u>Test Score ,Perplexity Calculation:</u>
For computing the Test Score
Can you write Python code for it?

#computes the score for test data i,e no. of right predictions against no. of wrong predictions

#return:int
#arg:list,dict,dict,dict,dict
def computeTestScore(test_token,bi_dict, tri_dict, quad_dict, vocab_dict,
                     bi_prob_dict, tri_prob_dict, quad_prob_dict, token_len,param):

    #increment the score value if correct prediction is made else decrement its value
    score = 0
    wrong = 0
    total = 0
    with open('Test_Scores/add1_smoothing_score.txt','w') as w:
        for sent in test_token:
            sen_token = sent[:3]
            sen = " ".join(sen_token)
            correct_word = sent[3]
            #select probable word candidates for prediction
            word_choice = chooseWords(sen, bi_prob_dict, tri_prob_dict, quad_prob_dict)
            result = doInterpolatedPredictionAdd1(sen, bi_dict, tri_dict, quad_dict, vocab_dict,token_len, word_choice, param)

            if result == correct_word:
                score+=1
            else:
                wrong += 1

            total += 1
        w.write('Total Word Prdictions: '+str(total) + '\n' +'Correct Prdictions: '+str(score) +
                '\n'+'Wrong Prdictions: '+str(wrong) + '\n'+'ACCURACY: '+str((score/total)*100)+'%' )
    #print stats
    print('Total Word Prdictions: '+str(total) + '\n' +'Correct Prdictions: '+str(score) +
                '\n'+'Wrong Prdictions: '+str(wrong) + '\n'+'ACCURACY:'+str((score/total)*100) )
    return score


I am working on the file "Notebooks/Word_Prediction_Add-1_Smoothing_with_Interpolation.ipynb".
The first task is:
<u>Word Prediction related Driver functions</u>
For choosing prediction word candidates for Word Prediction
Can you write Python code for it?

#pick the top most probable words from bi,tri and quad prob dict as word prediction candidates

#returns: list[float,string]
#arg: string,dict,dict,dict
def chooseWords(sen, bi_prob_dict, tri_prob_dict, quad_prob_dict):
    word_choice = []
    token = sen.split()
    if token[-1] in bi_prob_dict:
        word_choice +=  bi_prob_dict[token[-1]][:1]
        #print('Word Choice bi dict')
    if ' '.join(token[1:]) in tri_prob_dict:
        word_choice +=  tri_prob_dict[' '.join(token[1:])][:1]
        #print('Word Choice tri_dict')
    if ' '.join(token) in quad_prob_dict:
        word_choice += quad_prob_dict[' '.join(token)][:1]
        #print('Word Choice quad_dict')

    return word_choice


I am working on the file "Notebooks/Word_Prediction_Add-1_Smoothing_with_Interpolation.ipynb".
The first task is:
Finds the Predicted Word using Interpolation with Add -1 Smoothing
Can you write Python code for it?

#does prediction for the the sentence using Interpolation
#Uses Add-1 Smoothing
#returns: string
#arg: string,dict,dict,dict,dict,int,list,list
def doInterpolatedPredictionAdd1(sen, bi_dict, tri_dict, quad_dict,
                             vocab_dict,token_len, word_choice, param):
    pred = ''
    max_prob = 0.0
    V = len(vocab_dict)
    #for each word choice find the interpolated probability and decide
    for word in word_choice:
        key = sen + ' ' + word[1]
        quad_token = key.split()

        prob = (
                  param[0]*((quad_dict[key] + 1)/ (tri_dict[' '.join(quad_token[0:3])] + V))
                + param[1]*((tri_dict[' '.join(quad_token[1:4])] + 1) / (bi_dict[' '.join(quad_token[1:3])] + V))
                + param[2]*((bi_dict[' '.join(quad_token[2:4])] + 1) / (vocab_dict[quad_token[2]] + V))
                + param[3]*((vocab_dict[quad_token[3]] + 1) / (token_len + V))
               )

        if prob > max_prob:
            max_prob = prob
            pred = word
    #return only pred to get word with its prob
    if pred:
        return pred[1]
    else:
        return ''


I am working on the file "Notebooks/Word_Prediction_Add-1_Smoothing_with_Interpolation.ipynb".
The first task is:
For creating Probability Dictionary for Trigram Probabilities
Can you write Python code for it?

#for creating prob dict for trigram probabilities
#creates dict for storing probable words with their probabilities for a trigram sentence
# ADD 1 Smoothing used

#returns: void
#arg: dict,dict,dict,dict
def findTrigramProbAdd1(vocab_dict, bi_dict, tri_dict, tri_prob_dict):

    #vocabulary length
    V = len(vocab_dict)

    #create a dictionary of probable words with their probabilities for
    #trigram probabilites,key is a bigram and value is a list of prob and word
    for tri in tri_dict:
        tri_token = tri.split()
        #bigram sentence for key
        bi_sen = ' '.join(tri_token[:2])

        #find the probability
        #add 1 smoothing has been used
        prob = ( tri_dict[tri] + 1 ) / ( bi_dict[bi_sen] + V)

        #tri_prob_dict is a dict of list
        #if the bigram sentence is not present in the Dictionary then add it
        if bi_sen not in tri_prob_dict:
            tri_prob_dict[bi_sen] = []
            tri_prob_dict[bi_sen].append([prob,tri_token[-1]])
        #the bigram sentence is present but the probable word is missing,then add it
        else:
            tri_prob_dict[bi_sen].append([prob,tri_token[-1]])

    prob = None
    tri_token = None
    bi_sen = None


I am working on the file "Notebooks/Word_Prediction_Add-1_Smoothing_with_Interpolation.ipynb".
The first task is:
Sort the probable words
Can you write Python code for it?

#for sorting the probable word acc. to their probabilities

#returns: void
#arg: dict, dict, dict
def sortProbWordDict(bi_prob_dict, tri_prob_dict, quad_prob_dict):
    #sort bigram dict
    for key in bi_prob_dict:
        if len(bi_prob_dict[key])>1:
            bi_prob_dict[key] = sorted(bi_prob_dict[key],reverse = True)

    #sort trigram dict
    for key in tri_prob_dict:
        if len(tri_prob_dict[key])>1:
            tri_prob_dict[key] = sorted(tri_prob_dict[key],reverse = True)

    #sort quadgram dict
    for key in quad_prob_dict:
        if len(quad_prob_dict[key])>1:
            quad_prob_dict[key] = sorted(quad_prob_dict[key],reverse = True)[:2]


I am working on the file "Notebooks/Word_Prediction_Add-1_Smoothing_with_Interpolation.ipynb".
The first task is:
Tokenize and load the corpus data
Can you write Python code for it?

#returns : void
#arg: string,dict,dict,dict,dict
#loads the corpus for the dataset and makes the frequency count of quadgram ,bigram and trigram strings
def loadCorpus(file_path, bi_dict, tri_dict, quad_dict, vocab_dict):

    w1 = ''    #for storing the 3rd last word to be used for next token set
    w2 = ''    #for storing the 2nd last word to be used for next token set
    w3 = ''    #for storing the last word to be used for next token set
    token = []
    #total no. of words in the corpus
    word_len = 0

    #open the corpus file and read it line by line
    with open(file_path,'r') as file:
        for line in file:

            #split the string into word tokens
            temp_l = line.split()
            i = 0
            j = 0

            #does the same as the removePunctuations() function,implicit declaration for performance reasons
            #changes the word to lowercase and removes punctuations from it
            for word in temp_l :
                j = 0
                #print(len(word))
                for l in word :
                    if l in string.punctuation:
                        if l == "'":
                            if j+1<len(word) and word[j+1] == 's':
                                j = j + 1
                                continue
                        word = word.replace(l," ")
                        #print(j,word[j])
                    j += 1

                temp_l[i] = word.lower()
                i=i+1

            #spliting is being done here beacause in sentences line here---so after punctuation removal it should
            #become "here so"
            content = " ".join(temp_l)

            token = content.split()
            word_len = word_len + len(token)

            if not token:
                continue

            #add the last word from previous line
            if w3!= '':
                token.insert(0,w3)

            temp0 = list(ngrams(token,2))

            #since we are reading line by line some combinations of word...


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Backoff.ipynb".
The first task is:
For Debugging Purpose Only
Uncomment the above two cells and ignore running the cells below if not debugging
Can you write Python code for it?

#variable declaration
vocab_dict = defaultdict(int)          #for storing the different words with their frequencies
bi_dict = defaultdict(int)             #for keeping count of sentences of two words
tri_dict = defaultdict(int)            #for keeping count of sentences of three words
quad_dict = defaultdict(int)           #for keeping count of sentences of four words
quad_prob_dict = OrderedDict()
tri_prob_dict = OrderedDict()
bi_prob_dict = OrderedDict()

#load the corpus for the dataset
#loadCorpus('corpusfile.txt',bi_dict,tri_dict,quad_dict,vocab_dict)
print("---Preprocessing Time for Corpus loading: %s seconds ---" % (time.time() - start_time))


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Backoff.ipynb".
The first task is:
<u>Driver function for doing the prediction</u>
Find word Prediction using Backoff
Can you write Python code for it?

#finds the word prediction usinng Backoff
def doPredictionBackoffGT(input_sen, bi_dict, tri_dict, quad_dict, bi_prob_dict, tri_prob_dict, quad_prob_dict):
    #split the input sentence into tokens
    token = input_sen.split()

    #if the input sen is found in any ngram then give the most probable word for that ngram
    #if not then go to the lower order ngram
    if input_sen in quad_prob_dict and quad_prob_dict[ input_sen ][0][0]>0:
        pred = quad_prob_dict[input_sen][0]
    elif ' '.join(token[1:]) in tri_prob_dict and tri_prob_dict[' '.join(token[1:])][0][0]>0:
        pred = tri_prob_dict[ ' '.join(token[1:]) ][0]
    elif ' '.join(token[2:]) in bi_prob_dict and bi_prob_dict[ ' '.join(token[2:]) ][0][0]>0:
        pred = bi_prob_dict[' '.join(token[2:])][0]
    else:
        pred = []
    return pred


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Backoff.ipynb".
The first task is:
<u>Test Score ,Perplexity Calculation:</u>
For computing the Test Score
Can you write Python code for it?

#computes the score for test data
def computeTestScore(test_token, bi_dict, tri_dict, quad_dict,
                              quad_prob_dict, tri_prob_dict,bi_prob_dict ):
     #increment the score value if correct prediction is made else decrement its value
    score = 0
    wrong = 0
    total = 0
    with open('Test_Scores/Good_Turing_Backoff_Score.txt','w') as w:
        for sent in test_token:
            sen_token = sent[:3]
            sen = " ".join(sen_token)
            correct_word = sent[3]

            result = doPredictionBackoffGT(sen, bi_dict, tri_dict, quad_dict, bi_prob_dict, tri_prob_dict,
                                           quad_prob_dict)
            if result:
                if result[1] == correct_word:
                    score+=1
                else:
                    wrong += 1
            else:
                wrong += 1
            total += 1

        w.write('Total Word Prdictions: '+str(total) + '\n' +'Correct Prdictions: '+str(score) +
                '\n'+'Wrong Prdictions: '+str(wrong) + '\n'+'ACCURACY: '+str((score/total)*100)+'%' )
        #print stats
        print('Total Word Prdictions: '+str(total) + '\n' +'Correct Prdictions: '+str(score) +
                '\n'+'Wrong Prdictions: '+str(wrong) + '\n'+'ACCURACY:'+str((score/total)*100)+'%' )
    return score


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Backoff.ipynb".
The first task is:
Create a Hash Table for Probable words for Unigram
Can you write Python code for it?

#returns: void
#arg: dict,dict,dict,dict,int
#creates dict for storing probable words with their probabilities for a unigram
def findBigramProbGT(vocab_dict, bi_dict, bi_prob_dict, nc_dict, k):

    #vocabulary size
    V = len(vocab_dict)

    #create a dictionary of probable words with their probabilities for bigram probabilites
    for bi in bi_dict:
        bi_token = bi.split()
        #unigram for key
        unigram = bi_token[0]

        #find the probability
        #Good Turing smoothing has been used
        bi_count = bi_dict[bi]
        uni_count = vocab_dict[unigram]

        if bi_dict[bi] <= k or (bi not in bi_dict):
            bi_count = findGoodTuringAdjustCount( bi_dict[bi], k, nc_dict)
        if vocab_dict[unigram] <= k or (unigram not in vocab_dict):
            uni_count = findGoodTuringAdjustCount( vocab_dict[unigram], k, nc_dict)

        prob = bi_count / uni_count

        #add the unigram to the bigram probability dict
        #bi_prob_dict is a dict of list
        if unigram not in bi_prob_dict:
            bi_prob_dict[unigram] = []
            bi_prob_dict[unigram].append([prob,bi_token[-1]])
        else:
            bi_prob_dict[unigram].append([prob,bi_token[-1]])

    prob = None
    bi_token = None
    unigram = None


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Backoff.ipynb".
The first task is:
For Computing the Perplexity
Can you write Python code for it?

#return:float
#arg:list,int,dict,dict,dict,dict
#computes the score for test data
def computePerplexity(test_quadgrams, bi_dict, tri_dict, quad_dict,
                                     vocab_dict,token_len, k, quad_nc_dict, tri_nc_dict,
                                      bi_nc_dict, uni_nc_dict):

    perplexity = float(1.0)
    n = token_len

    for key in quad_dict:
        quad_token = key.split()

        quad_count = quad_dict[key]
        tri_count = tri_dict[' '.join(quad_token[0:3])]

        if quad_dict[key] <= k or (key not in quad_dict):
            quad_count = findGoodTuringAdjustCount( quad_dict[key], k, quad_nc_dict)
        if tri_dict[' '.join(quad_token[0:3])] <= k  or (' '.join(quad_token[0:3]) not in tri_dict):
            tri_count = findGoodTuringAdjustCount( tri_dict[' '.join(quad_token[0:3])], k, tri_nc_dict)
        prob = quad_count / tri_count

        if prob != 0:
            perplexity = perplexity * ( prob**(1./n))
    with open('Test_Scores/Good_Turing_Backoff_Score.txt','a') as w:
        w.write('\nPerplexity: '+str(perplexity))
    return perplexity


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Backoff.ipynb".
The first task is:
For Testing the Language Model
Calculates % Accuracy and Perplexity<br>
NOTE : If this is run then no need to run the cells following it
Can you write Python code for it?

train_file = 'training_corpus.txt'
test_file = 'test_corpus.txt'
#load the corpus for the dataset
token_len = trainCorpus(train_file,test_file,bi_dict,tri_dict,quad_dict,vocab_dict,quad_prob_dict)

train_file = 'corpusfile.txt'
#load corpus
token_len = loadCorpus(train_file, bi_dict, tri_dict, quad_dict, vocab_dict)

#create the different Nc dictionaries for ngrams
#threshold value
k = 5
V = len(vocab_dict)
quad_nc_dict = findFrequencyOfFrequencyCount(quad_dict, k, 4, V, len(quad_dict))
tri_nc_dict = findFrequencyOfFrequencyCount(tri_dict, k, 3, V, len(tri_dict))
bi_nc_dict = findFrequencyOfFrequencyCount(bi_dict, k, 2, V, len(bi_dict))
uni_nc_dict = findFrequencyOfFrequencyCount(bi_dict, k, 1, V, len(vocab_dict))

#create quadgram probability dictionary
findQuadgramProbGT(vocab_dict, bi_dict, tri_dict, quad_dict, quad_prob_dict, quad_nc_dict, k)

#create trigram probability dictionary
findTrigramProbGT(vocab_dict, bi_dict, tri_dict, tri_prob_dict, tri_nc_dict, k)

#create bigram probability dictionary
findBigramProbGT(vocab_dict, bi_dict, bi_prob_dict, bi_nc_dict, k)

#sort the probability dictionaries of quad,tri and bi grams
sortProbWordDict(bi_prob_dict, tri_prob_dict, quad_prob_dict)

#FOR DEBUGGING ONLY
writeProbDicts(bi_prob_dict, tri_prob_dict, quad_prob_dict)

##WORD PREDICTION

start_time2 = time.time()
#take user input
input_sen = takeInput()

prediction = doPredictionBackoffGT(input_sen, bi_dict, tri_dict, quad_dict, bi_prob_dict, tri_prob_dict, quad_prob_dict)
if prediction:
    print('Word Prediction:',prediction[1])
print("---Time for Prediction Operation: %s seconds ---" % (time.time() - start_time2))


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Backoff.ipynb".
The first task is:
Sort the probable words for the various Probability Dictionaries according to their probability
Can you write Python code for it?

#returns: void
#arg: dict
#for sorting the probable word acc. to their probabilities
def sortProbWordDict(bi_prob_dict, tri_prob_dict, quad_prob_dict):
    for key in bi_prob_dict:
        if len(bi_prob_dict[key])>1:
            bi_prob_dict[key] = sorted(bi_prob_dict[key],reverse = True)

    for key in tri_prob_dict:
        if len(tri_prob_dict[key])>1:
            tri_prob_dict[key] = sorted(tri_prob_dict[key],reverse = True)

    for key in quad_prob_dict:
        if len(quad_prob_dict[key])>1:
            quad_prob_dict[key] = sorted(quad_prob_dict[key],reverse = True)[:2]


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Backoff.ipynb".
The first task is:
Tokenize and load the corpus data
Can you write Python code for it?

#returns : int
#arg: string,dict,dict,dict,dict
#loads the corpus for the dataset and makes the frequency count of quadgram ,bigram and trigram strings
def loadCorpus(file_path, bi_dict, tri_dict, quad_dict, vocab_dict):

    w1 = ''    #for storing the 3rd last word to be used for next token set
    w2 = ''    #for storing the 2nd last word to be used for next token set
    w3 = ''    #for storing the last word to be used for next token set
    token = []
    #total no. of words in the corpus
    word_len = 0

    #open the corpus file and read it line by line
    with open(file_path,'r') as file:
        for line in file:

            #split the string into word tokens
            temp_l = line.split()
            i = 0
            j = 0

            #does the same as the removePunctuations() function,implicit declratation for performance reasons
            #changes the word to lowercase and removes punctuations from it
            for word in temp_l :
                j = 0
                #print(len(word))
                for l in word :
                    if l in string.punctuation:
                        if l == "'":
                            if j+1<len(word) and word[j+1] == 's':
                                j = j + 1
                                continue
                        word = word.replace(l," ")
                        #print(j,word[j])
                    j += 1

                temp_l[i] = word.lower()
                i=i+1

            #spliting is being done here beacause in sentences line here---so after punctuation removal it should
            #become "here so"
            content = " ".join(temp_l)

            token = content.split()
            word_len = word_len + len(token)

            if not token:
                continue

            #add the last word from previous line
            if w3!= '':
                token.insert(0,w3)

            temp0 = list(ngrams(token,2))

            #since we are reading line by line some combinations of word...


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Backoff.ipynb".
The first task is:
main function
Can you write Python code for it?

def main():
    #variable declaration
    vocab_dict = defaultdict(int)          #for storing the different words with their frequencies
    bi_dict = defaultdict(int)             #for keeping count of sentences of two words
    tri_dict = defaultdict(int)            #for keeping count of sentences of three words
    quad_dict = defaultdict(int)           #for keeping count of sentences of four words
    quad_prob_dict = OrderedDict()
    tri_prob_dict = OrderedDict()
    bi_prob_dict = OrderedDict()

    #load the corpus for the dataset
    train_file = 'corpusfile.txt'
    #load corpus
    token_len = loadCorpus(train_file, bi_dict, tri_dict, quad_dict, vocab_dict)

    #create the different Nc dictionaries for ngrams
    #threshold value
    k = 5
    V = len(vocab_dict)
    quad_nc_dict = findFrequencyOfFrequencyCount(quad_dict, k, 4, V, len(quad_dict))
    tri_nc_dict = findFrequencyOfFrequencyCount(tri_dict, k, 3, V, len(tri_dict))
    bi_nc_dict = findFrequencyOfFrequencyCount(bi_dict, k, 2, V, len(bi_dict))
    uni_nc_dict = findFrequencyOfFrequencyCount(bi_dict, k, 1, V, len(vocab_dict))

    #create quadgram probability dictionary
    findQuadgramProbGT(vocab_dict, bi_dict, tri_dict, quad_dict, quad_prob_dict, quad_nc_dict, k)
    #create trigram probability dictionary
    findTrigramProbGT(vocab_dict, bi_dict, tri_dict, tri_prob_dict, tri_nc_dict, k)
    #create bigram probability dictionary
    findBigramProbGT(vocab_dict, bi_dict, bi_prob_dict, bi_nc_dict, k)
    #sort the probability dictionaries of quad,tri and bi grams
    sortProbWordDict(bi_prob_dict, tri_prob_dict, quad_prob_dict)

    ##WORD PREDICTION
    #take user input
    input_sen = takeInput()

    prediction = doPredictionBackoffGT(input_sen, bi_dict, tri_dict, quad_dict, bi_prob_dict, tri_prob_dict, quad_prob_dict)
    if prediction:
        print('Word Prediction:',prediction[1])

if __name__ == '__main__':
    main()


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
For Debugging Purpose Only
Uncomment the above two cells and ignore running the cells below if not debugging
Can you write Python code for it?

#variable declaration
vocab_dict = defaultdict(int)          #for storing the different words with their frequencies
bi_dict = defaultdict(int)             #for keeping count of sentences of two words
tri_dict = defaultdict(int)            #for keeping count of sentences of three words
quad_dict = defaultdict(int)           #for keeping count of sentences of four words
quad_prob_dict = OrderedDict()
tri_prob_dict = OrderedDict()
bi_prob_dict = OrderedDict()

print("---Preprocessing Time for Corpus loading: %s seconds ---" % (time.time() - start_time))


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
<u>Do preprocessing</u>:
Remove the punctuations and lowercase the tokens
Can you write Python code for it?

#returns: string
#arg: string
#remove punctuations and make the string lowercase
def removePunctuations(sen):
    #split the string into word tokens
    temp_l = sen.split()
    #print(temp_l)
    i = 0
    j = 0

    #changes the word to lowercase and removes punctuations from it
    for word in temp_l :
        j = 0
        #print(len(word))
        for l in word :
            if l in string.punctuation:
                if l == "'":
                    if j+1<len(word) and word[j+1] == 's':
                        j = j + 1
                        continue
                word = word.replace(l," ")
                #print(j,word[j])
            j += 1

        temp_l[i] = word.lower()
        i=i+1

    #spliting is being don here beacause in sentences line here---so after punctuation removal it should
    #become "here so"
    content = " ".join(temp_l)

    return content


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
<u>Driver Function for Testing the Language Model</u>
Can you write Python code for it?

#return: void
#arg:string,string,dict,dict,dict,dict,dict
#Used for testing the Language Model
def trainCorpus(train_file,test_file,bi_dict,tri_dict,quad_dict,vocab_dict,prob_dict):

    test_result = ''
    score = 0
    #load the training corpus for the dataset
    token_len = loadCorpus(train_file, bi_dict, tri_dict, quad_dict, vocab_dict)
    print("---Processing Time for Corpus Loading: %s seconds ---" % (time.time() - start_time))

    start_time1 = time.time()

    #create the different Nc dictionaries for ngrams
    #threshold value
    k = 5
    V = len(vocab_dict)
    quad_nc_dict = findFrequencyOfFrequencyCount(quad_dict, k, 4, V, len(quad_dict))
    tri_nc_dict = findFrequencyOfFrequencyCount(tri_dict, k, 3, V, len(tri_dict))
    bi_nc_dict = findFrequencyOfFrequencyCount(bi_dict, k, 2, V, len(bi_dict))
    uni_nc_dict = findFrequencyOfFrequencyCount(bi_dict, k, 1, V, len(vocab_dict))

    #create quadgram probability dictionary
    findQuadgramProbGT(vocab_dict, bi_dict, tri_dict, quad_dict, quad_prob_dict, quad_nc_dict, k)
    #create trigram probability dictionary
    findTrigramProbGT(vocab_dict, bi_dict, tri_dict, tri_prob_dict, tri_nc_dict, k)
    #create bigram probability dictionary
    findBigramProbGT(vocab_dict, bi_dict, bi_prob_dict, bi_nc_dict, k)
    #sort the probability dictionaries of quad,tri and bi grams
    sortProbWordDict(bi_prob_dict, tri_prob_dict, quad_prob_dict)
    #Do only when required to find the lambda value as this can take some time
    #param = estimateParameters(token_len, vocab_dict, bi_dict, tri_dict, quad_dict)
    #found earlier using Held out data
    param = [0.0,0.0,0.7999999999999999,0.20000000000000007]
    print("---Processing Time for Creating Probable Word Dict: %s seconds ---" % (time.time() - start_time1))

    ### TESTING WITH TEST CORPUS
    test_data = ''
    #Now load the test corpus
    with open('test_corpus.txt','r') as file :
        test_data = file.read()

    #remove punctuations from the test data
    test_data =...


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
<u>Driver function for doing the prediction</u>
Do word Prediction using Interpolation
Can you write Python code for it?

#For doing word prediction using Interpolation
def doInterpolatedPredictionGT(sen, bi_dict, tri_dict, quad_dict,
                             vocab_dict,token_len, word_choice, param, k, quad_nc_dict, tri_nc_dict,
                              bi_nc_dict, uni_nc_dict ):

    pred = ''
    max_prob = 0.0
    V = len(vocab_dict)
    #for each word choice find the interpolated probability and decide
    for word in word_choice:
        key = sen + ' ' + word[1]
        quad_token = key.split()

        #find the Good Turing probabilty for quadgram probability
        quad_count = quad_dict[key]
        tri_count = tri_dict[' '.join(quad_token[0:3])]

        if quad_dict[key] <= k or (key not in quad_dict):
            quad_count = findGoodTuringAdjustCount( quad_dict[key], k, quad_nc_dict)
        if tri_dict[' '.join(quad_token[0:3])] <= k  or (' '.join(quad_token[0:3]) not in tri_dict):
            tri_count = findGoodTuringAdjustCount( tri_dict[' '.join(quad_token[0:3])], k, tri_nc_dict)
        quad_prob = quad_count / tri_count

        #find the Good Turing probabilty for trigram probability
        tri_count = tri_dict[' '.join(quad_token[1:4])]
        bi_count = bi_dict[' '.join(quad_token[1:3])]

        if tri_dict[' '.join(quad_token[1:4])] <= k  or (' '.join(quad_token[1:4]) not in tri_dict):
            tri_count = findGoodTuringAdjustCount( tri_dict[' '.join(quad_token[1:4])], k, tri_nc_dict)
        if bi_dict[' '.join(quad_token[1:3])] <= k or (' '.join(quad_token[1:3]) not in bi_dict):
            bi_count = findGoodTuringAdjustCount( bi_dict[' '.join(quad_token[1:3])], k, bi_nc_dict)
        tri_prob = tri_count / bi_count

        #find the Good Turing probabilty for bigram probability
        bi_count = bi_dict[' '.join(quad_token[2:4])]
        uni_count = vocab_dict[quad_token[2]]

        if bi_dict[' '.join(quad_token[2:4])] <= k or (' '.join(quad_token[2:4]) not in bi_dict):
            bi_count = findGoodTuringAdjustCount(...


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
<u>Test Score ,Perplexity Calculation:</u>
For computing the Test Score
Can you write Python code for it?

#computes the score for test data
def computeTestScore(test_token, bi_dict, tri_dict, quad_dict,
                             vocab_dict,token_len, param, k, quad_nc_dict, tri_nc_dict,
                              bi_nc_dict, uni_nc_dict ):
     #increment the score value if correct prediction is made else decrement its value
    score = 0
    wrong = 0
    total = 0
    with open('Test_Scores/Good_Turing_Interpolated_Score.txt','w') as w:
        for sent in test_token:
            sen_token = sent[:3]
            sen = " ".join(sen_token)
            correct_word = sent[3]
            #find the the most probable words for the bigram, trigram and unigram  sentence
            word_choice = chooseWords(sen, bi_prob_dict, tri_prob_dict, quad_prob_dict)

            result = doInterpolatedPredictionGT(sen, bi_dict, tri_dict, quad_dict,
                             vocab_dict,token_len, word_choice, param, k, quad_nc_dict, tri_nc_dict,
                              bi_nc_dict, uni_nc_dict )
            if result:
                if result[1] == correct_word:
                    score+=1
                else:
                    wrong += 1
            else:
                wrong += 1
            total += 1

        w.write('Total Word Prdictions: '+str(total) + '\n' +'Correct Prdictions: '+str(score) +
                '\n'+'Wrong Prdictions: '+str(wrong) + '\n'+'ACCURACY: '+str((score/total)*100)+'%' )
        #print stats
        print('Total Word Prdictions: '+str(total) + '\n' +'Correct Prdictions: '+str(score) +
                '\n'+'Wrong Prdictions: '+str(wrong) + '\n'+'ACCURACY:'+str((score/total)*100)+'%' )
    return score


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
Create a Hash Table for Probable words for Trigram sentences
Can you write Python code for it?

#returns: void
#arg: dict,dict,dict,dict,dict,dict,int
#creates dict for storing probable words with their probabilities for a trigram sentence
def findQuadgramProbGT(vocab_dict, bi_dict, tri_dict, quad_dict, quad_prob_dict, nc_dict, k):

    i = 0
    V = len(vocab_dict)

    for quad_sen in quad_dict:
        quad_token = quad_sen.split()

        #trigram sentence for key
        tri_sen = ' '.join(quad_token[:3])

        #find the probability
        #Good Turing smoothing has been used
        quad_count = quad_dict[quad_sen]
        tri_count = tri_dict[tri_sen]

        if quad_dict[quad_sen] <= k  or (quad_sen not in quad_dict):
            quad_count = findGoodTuringAdjustCount( quad_dict[quad_sen], k, nc_dict)
        if tri_dict[tri_sen] <= k  or (tri_sen not in tri_dict):
            tri_count = findGoodTuringAdjustCount( tri_dict[tri_sen], k, nc_dict)

        prob = quad_count / tri_count

        #add the trigram to the quadgram probabiltity dict
        if tri_sen not in quad_prob_dict:
            quad_prob_dict[tri_sen] = []
            quad_prob_dict[tri_sen].append([prob,quad_token[-1]])
        else:
            quad_prob_dict[tri_sen].append([prob,quad_token[-1]])

    prob = None
    quad_token = None
    tri_sen = None


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
Create a Hash Table for Probable words for Unigram
Can you write Python code for it?


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
For Testing the Language Model
Calculates % Accuracy and Perplexity<br>
NOTE : If this is run then no need to run the cells following it
Can you write Python code for it?

train_file = 'training_corpus.txt'
test_file = 'test_corpus.txt'
#load the corpus for the dataset
token_len = trainCorpus(train_file,test_file,bi_dict,tri_dict,quad_dict,vocab_dict,quad_prob_dict)

train_file = 'corpusfile.txt'
#load the corpus for the dataset
token_len = loadCorpus(train_file, bi_dict, tri_dict, quad_dict, vocab_dict)

#create the different Nc dictionaries for ngrams
#threshold value
k = 5
V = len(vocab_dict)
quad_nc_dict = findFrequencyOfFrequencyCount(quad_dict, k, 4, V, len(quad_dict))
tri_nc_dict = findFrequencyOfFrequencyCount(tri_dict, k, 3, V, len(tri_dict))
bi_nc_dict = findFrequencyOfFrequencyCount(bi_dict, k, 2, V, len(bi_dict))
uni_nc_dict = findFrequencyOfFrequencyCount(bi_dict, k, 1, V, len(vocab_dict))

#create quadgram probability dictionary
findQuadgramProbGT(vocab_dict, bi_dict, tri_dict, quad_dict, quad_prob_dict, quad_nc_dict, k)

#create trigram probability dictionary
findTrigramProbGT(vocab_dict, bi_dict, tri_dict, tri_prob_dict, tri_nc_dict, k)

#create bigram probability dictionary
findBigramProbGT(vocab_dict, bi_dict, bi_prob_dict, bi_nc_dict, k)

#sort the probability dictionaries of quad,tri and bi grams
sortProbWordDict(bi_prob_dict, tri_prob_dict, quad_prob_dict)

#Do only when required to find the lambda value as this can take some time
#param = estimateParameters(token_len, vocab_dict, bi_dict, tri_dict, quad_dict)
#found earlier using Held out data
param = [0.7,0.1,0.1,0.1]

#FOR DEBUGGING ONLY
writeProbDicts(bi_prob_dict, tri_prob_dict, quad_prob_dict)

##WORD PREDICTION

start_time2 = time.time()
#take user input
input_sen = takeInput()
#find the the most probable words for the bigram, trigram and unigram  sentence
word_choice = chooseWords(input_sen, bi_prob_dict, tri_prob_dict, quad_prob_dict)

prediction = doInterpolatedPredictionGT(input_sen, bi_dict, tri_dict, quad_dict,
                             vocab_dict,token_len, word_choice, param, k, quad_nc_dict, tri_nc_dict,
                              bi_nc_dict, uni_nc_dict )
if...


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
For choosing Probable words as Word Prediction candidate
Can you write Python code for it?

#pick the top most probable words from bi,tri and quad prob dict as word prediction candidates

#returns: list[float,string]
#arg: string,dict,dict,dict
def chooseWords(sen, bi_prob_dict, tri_prob_dict, quad_prob_dict):
    word_choice = []
    token = sen.split()
    if token[-1] in bi_prob_dict:
        word_choice +=  bi_prob_dict[token[-1]][:1]
        #print('Word Choice bi dict')
    if ' '.join(token[1:]) in tri_prob_dict:
        word_choice +=  tri_prob_dict[' '.join(token[1:])][:1]
        #print('Word Choice tri_dict')
    if ' '.join(token) in quad_prob_dict:
        word_choice += quad_prob_dict[' '.join(token)][:1]
        #print('Word Choice quad_dict')

    return word_choice


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
Parameter estimation
For estimating parameters we try to maximise the value of lambdas l1,l2,l3 and l4

We do that by try all possible combinations of lambdas with step size 0.1 and try to maximise the

 probabilty of held out data
Can you write Python code for it?

#finds the lambda values required for doing Interpolation

#arg: int, dict, dict, dict, dict
#returns: list
def estimateParameters(token_len, vocab_dict, bi_dict, tri_dict, quad_dict):
    max_prob = -9999999999999999999.0
    curr_prob = 0.0
    parameters = [0.0,0.0,0.0,0.0]
    i = 1

    #load the held out data
    file = open('held_out_corpus.txt','r')
    held_out_data = file.read()
    file.close()

    #remove punctuations and other cleaning stuff
    held_out_data = removePunctuations(held_out_data)
    held_out_data = held_out_data.split()
    #make quad tokens for parameter estimation
    quad_token_heldout = list(ngrams(held_out_data,4))

    #for storing the stats
    f = open('interpolation_prob_stats.txt','w')

    #lambda values1 and 4
    l1 = 0
    l4 = 0

    while l1 <= 1.0:
        l2 = 0
        while l2 <= 1.0:
            l3 = 0
            while l3 <= 1.0:

                #when the sum of lambdas is greater than 1 or when all 4 are zero we don't need to check so skip
                if l1 == 0 and l2 == 0 and l3 == 0 or ((l1+l2+l3)>1):
                    l3 += 0.1
                    i += 1
                    continue

                #find lambda 4
                l4 = 1- (l1 + l2 + l3)

                curr_prob = 0
                qc = [0]
                bc = [0]
                tc = [0]

                #find the probability for the held out set using the current lambda values
                for quad in quad_token_heldout:
                    #take log of prob to avoid underflow
                    curr_prob += log10( interpolatedProbability(quad,token_len, vocab_dict, bi_dict, tri_dict,
                                                                quad_dict,qc,bc,tc,l1, l2, l3, l4) )

                if curr_prob > max_prob:
                    max_prob = curr_prob
                    parameters[0] = l1
                    parameters[1] = l2
                   ...


I am working on the file "Notebooks/Word_Prediction_GoodTuring_Smoothing_with_Interpolation.ipynb".
The first task is:
Sort the probable words for the various Probability Dictionaries according to their probability
Can you write Python code for it?


I am working on the file "Notebooks/Word_Prediction_Quadgram_In_Constant_Time.ipynb".
The first task is:
For Debugging Purpose Only
Uncomment the above two cells and ignore running the cells below if not debugging
Can you write Python code for it?

#variable declaration
tri_dict = defaultdict(int)            #for keeping count of sentences of three words
quad_dict = defaultdict(int)           #for keeping count of sentences of three words
vocab_dict = defaultdict(int) #for storing the different words with their frequencies
prob_dict = OrderedDict()   #for storing the probabilities of probable words for a sentence
bi_dict = defaultdict(int)

#load the corpus for the dataset
token_len = loadCorpus('corpusfile.txt',bi_dict,tri_dict,quad_dict,vocab_dict)
print("---Preprocessing Time for Corpus loading: %s seconds ---" % (time.time() - start_time))

start_time1 = time.time()

#creates a dictionary of probable words
createProbableWordDict(bi_dict,tri_dict,quad_dict,prob_dict,vocab_dict,token_len)
#sort the dictionary of probable words
sortProbWordDict(prob_dict)
# writeProbWords(prob_dict)
gc.collect()
print("---Preprocessing Time for Creating Probable Word Dict: %s seconds ---" % (time.time() - start_time1))

sen = takeInput()
start_time2 = time.time()
prediction = doPrediction(sen,prob_dict)
print("Word Prediction:",prediction)
print("---Time for Prediction Operation: %s seconds ---" % (time.time() - start_time2))


I am working on the file "Notebooks/Word_Prediction_Quadgram_In_Constant_Time.ipynb".
The first task is:
<u> For Computing Interpolated Probability</u>
Can you write Python code for it?

#returns: float
#arg: float,float,float,float,list,list,dict,dict,dict,dict
#for calculating the interpolated probablity
def interpolatedProbability(quad_token,token_len, vocab_dict, bi_dict, tri_dict, quad_dict,
                            l1 = 0.25, l2 = 0.25, l3 = 0.25 , l4 = 0.25):

    sen = ' '.join(quad_token)
    prob =(
              l1*(quad_dict[sen] / tri_dict[' '.join(quad_token[0:3])])
            + l2*(tri_dict[' '.join(quad_token[1:4])] / bi_dict[' '.join(quad_token[1:3])])
            + l3*(bi_dict[' '.join(quad_token[2:4])] / vocab_dict[quad_token[2]])
            + l4*(vocab_dict[quad_token[3]] / token_len)
          )
    return prob


I am working on the file "Notebooks/Word_Prediction_Quadgram_In_Constant_Time.ipynb".
The first task is:
<u>Do preprocessing</u>:
Remove the punctuations and lowercase the tokens
Can you write Python code for it?

#returns: string
#arg: string
#remove punctuations and make the string lowercase
def removePunctuations(sen):

    #split the string into word tokens
    temp_l = sen.split()
    i = 0

    #changes the word to lowercase and removes punctuations from it
    for word in temp_l :
        for l in word :
            if l in string.punctuation:
                word = word.replace(l," ")
        temp_l[i] = word.lower()
        i=i+1

    #spliting is being don here beacause in sentences line here---so after punctuation removal it should
    #become "here so"
    content = " ".join(temp_l)

    return content


I am working on the file "Notebooks/Word_Prediction_Quadgram_In_Constant_Time.ipynb".
The first task is:
<u>Driver function for doing the prediction</u>
Can you write Python code for it?

#returns: string
#arg: string,dict,int
#does prediction for the the sentence
def doPrediction(sen,prob_dict,rank = 1):
    if sen in prob_dict:
        if rank <= len(prob_dict[sen]):
            return prob_dict[sen][rank-1][1]
        else:
            return prob_dict[sen][0][1]
    else:
        return "Can't predict"


I am working on the file "Notebooks/Word_Prediction_Quadgram_In_Constant_Time.ipynb".
The first task is:
<u>For Taking input from the User</u>
Can you write Python code for it?

#returns: string
#arg: void
#for taking input from user
def takeInput():
    cond = False
    #take input
    while(cond == False):
        sen = input('Enter the string\n')
        sen = removePunctuations(sen)
        temp = sen.split()
        if len(temp) < 3:
            print("Please enter atleast 3 words !")
        else:
            cond = True
            temp = temp[-3:]
    sen = " ".join(temp)
    return sen


I am working on the file "Notebooks/Word_Prediction_Quadgram_In_Constant_Time.ipynb".
The first task is:
<u>main function</u>
Can you write Python code for it?

"""
def main():

    #variable declaration
    tri_dict = defaultdict(int)            #for keeping count of sentences of three words
    quad_dict = defaultdict(int)           #for keeping count of sentences of three words
    vocab_dict = defaultdict(int) #for storing the different words with their frequencies
    prob_dict = OrderedDict()   #for storing the probabilities of probable words for a sentence
    bi_dict = defaultdict(int)

    #load the corpus for the dataset
    token_len = loadCorpus('corpusfile.txt',bi_dict,tri_dict,quad_dict,vocab_dict)
    print("---Preprocessing Time for Corpus loading: %s seconds ---" % (time.time() - start_time))

    start_time1 = time.time()

    #creates a dictionary of probable words
    createProbableWordDict(bi_dict,tri_dict,quad_dict,prob_dict,vocab_dict,token_len)
    #sort the dictionary of probable words
    sortProbWordDict(prob_dict)
    # writeProbWords(prob_dict)
    gc.collect()
    print("---Preprocessing Time for Creating Probable Word Dict: %s seconds ---" % (time.time() - start_time1))
""""

"""
if __name__ == '__main__':
    main()
"""


I am working on the file "Notebooks/Word_Prediction_Quadgram_In_Constant_Time.ipynb".
The first task is:
Create a Hash Table for Probable words for Trigram sentences
Can you write Python code for it?

#returns: void
#arg: dict,dict,dict,dict,dict,int
#creates dict for storing probable words with their probabilities for a trigram sentence
def createProbableWordDict(bi_dict,tri_dict,quad_dict,prob_dict,vocab_dict,token_len):
    for quad_sen in quad_dict:
        prob = 0.0
        quad_token = quad_sen.split()
        tri_sen = ' '.join(quad_token[:3])
        tri_count = tri_dict[tri_sen]

        if tri_count != 0:
            prob = interpolatedProbability(quad_token,token_len, vocab_dict, bi_dict, tri_dict, quad_dict,
                            l1 = 0.25, l2 = 0.25, l3 = 0.25 , l4 = 0.25)

        if tri_sen not in prob_dict:
            prob_dict[tri_sen] = []
            prob_dict[tri_sen].append([prob,quad_token[-1]])
        else:
            prob_dict[tri_sen].append([prob,quad_token[-1]])

    prob = None
    tri_count = None
    quad_token = None
    tri_sen = None


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
For Debugging Purpose Only
Ignore running the cells below if not debugging
Can you write Python code for it?

#variable declaration
vocab_dict = defaultdict(int)          #for storing the different words with their frequencies
bi_dict = defaultdict(int)             #for keeping count of sentences of two words
tri_dict = defaultdict(int)            #for keeping count of sentences of three words
quad_dict = defaultdict(int)           #for keeping count of sentences of four words
prob_dict = defaultdict(list)         #for storing probability of probable words for prediction

#load the corpus for the dataset
#loadCorpus('corpusfile.txt',bi_dict,tri_dict,quad_dict,vocab_dict)
print("---Preprocessing Time for Corpus loading: %s seconds ---" % (time.time() - start_time))


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
<u>Do preprocessing</u>:
Remove the punctuations and lowercase the tokens
Can you write Python code for it?

#returns: string
#arg: string
#remove punctuations, change to lowercase ,retain the apostrophe mark
def removePunctuations(sen):
    #split the string into word tokens
    temp_l = sen.split()
    #print(temp_l)
    i = 0
    j = 0

    #changes the word to lowercase and removes punctuations from it
    for word in temp_l :
        j = 0
        #print(len(word))
        for l in word :
            if l in string.punctuation:
                if l == "'":
                    if j+1<len(word) and word[j+1] == 's':
                        j = j + 1
                        continue
                word = word.replace(l," ")
                #print(j,word[j])
            j += 1

        temp_l[i] = word.lower()
        i=i+1

    #spliting is being done here beacause in sentences line here---so after punctuation removal it should
    #become "here so"
    content = " ".join(temp_l)

    return content


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
<u>Driver Function for Testing the Language Model</u>
Can you write Python code for it?

#return: void
#arg:string,string,dict,dict,dict,dict,dict
#Used for testing the Language Model
def trainCorpus(train_file,test_file,bi_dict,tri_dict,quad_dict,vocab_dict,prob_dict):

    score = 0
    #load the training corpus for the dataset
    #token_len = loadCorpus1(bi_dict,tri_dict,quad_dict,vocab_dict)
    token_len = loadCorpus(train_file, bi_dict, tri_dict, quad_dict, vocab_dict)
    print("---Processing Time for Corpus Loading: %s seconds ---" % (time.time() - start_time))

    start_time1 = time.time()
    #compute the Knesser Ney probabilities
    computeKnesserNeyProb1(vocab_dict, bi_dict, tri_dict, quad_dict, prob_dict )
    #sort the probable words by their probability
    sortProbWordDict(prob_dict)
    gc.collect()
    print("---Preprocessing Time for Creating Probable Word Dict: %s seconds ---" % (time.time() - start_time1))

### TESTING WITH TEST CORPUS
    test_data = ''
    #Now load the test corpus
    #!!!!!!!!!!!!!ONLY FOR THIS CORPUS!!!!!!!!
    with open('test_corpus.txt','r') as file :
        for line in file:
            line_token = line.split()
            test_data = test_data + ' '+ ' '.join(line_token[1:])

    ### TESTING WITH TEST CORPUS
    test_data = ''
    #Now load the test corpus
    with open('test_corpus.txt','r') as file :
        test_data = file.read()

    #remove punctuations from the test data
    test_data = removePunctuations(test_data)
    test_token = test_data.split()

    #split the test data into 4 words list
    test_token = test_data.split()
    test_quadgrams = list(ngrams(test_token,4))

    #print(len(test_token))
    start_time1 = time.time()
    score = computeTestScore(test_quadgrams,tri_dict,quad_dict,vocab_dict,prob_dict)
    print('Score:',score)
    print("---Processing Time for computing score: %s seconds ---" % (time.time() - start_time1))

    start_time2 = time.time()
    perplexity = computePerplexity(test_token,token_len,tri_dict,quad_dict,vocab_dict,prob_dict)
    print('Perplexity:',perplexity)
   ...


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
<u>For Taking input from the User</u>
Can you write Python code for it?


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
<u>Test Score ,Perplexity Calculation:</u>
For computing the Test Score
Can you write Python code for it?

#return:int
#arg:list,dict,dict,dict,dict
#computes the score for test data
def computeTestScore(test_sent,tri_dict,quad_dict,vocab_dict,prob_dict):
    #increment the score value if correct prediction is made else decrement its value
    score = 0
    wrong = 0
    total = 0
    with open('Test_Scores/Knesser_Ney_Interpolated_Score.txt','w') as w:
        for sent in test_sent:
            sen_token = sent[:3]
            sen = " ".join(sen_token)
            correct_word = sent[3]

            result = doPrediction(sen,prob_dict)
            if result == correct_word:
                score+=1
            else:
                wrong += 1

            total += 1

        w.write('Total Word Prdictions: '+str(total) + '\n' +'Correct Prdictions: '+str(score) +
                '\n'+'Wrong Prdictions: '+str(wrong) + '\n'+'ACCURACY: '+str((score/total)*100)+'%' )
        #print stats
        print('Total Word Prdictions: '+str(total) + '\n' +'Correct Prdictions: '+str(score) +
                '\n'+'Wrong Prdictions: '+str(wrong) + '\n'+'ACCURACY:'+str((score/total)*100)+'%' )
    return score


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
<u>main function</u>
Can you write Python code for it?

def main():
#variable declaration
vocab_dict = defaultdict(int)          #for storing the different words with their frequencies
bi_dict = defaultdict(int)             #for keeping count of sentences of two words
tri_dict = defaultdict(int)            #for keeping count of sentences of three words
quad_dict = defaultdict(int)           #for keeping count of sentences of four words
prob_dict = defaultdict(list)         #for storing probability of probable words for prediction

train_file = 'corpusfile.txt'
token_len = loadCorpus(train_file, bi_dict, tri_dict, quad_dict, vocab_dict)

#compute the Knesser Ney probabilities for bigrams,trigrams and unigrams
computeKnesserNeyProb(vocab_dict, bi_dict, tri_dict, quad_dict, prob_dict )

#sort the probable words by their probability
sortProbWordDict(prob_dict)

#take user input
input_sen = takeInput()

### PREDICTION
prediction = doPrediction(input_sen,prob_dict)
print('Word Prediction:',prediction)

if __name__ == '__main__':
    main()


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
For Computing Knesser Ney probability for various words
Can you write Python code for it?

#Finds the Knesser Ney probability for prediction
#return:dict,dict
#arg:dict, int
def computeKnesserNeyProb(vocab_dict, bi_dict, tri_dict, quad_dict, prob_dict ):
        d = 0.75
        #first create the dict for storing the count of Wn-1 followed by Wn and for
        #ngrams preceding Wn-1

        #for quadgram
        quad_first_dict, quad_sec_dict = createKNDict(quad_dict, 4)

        #for trigram
        tri_first_dict, tri_sec_dict = createKNDict(tri_dict, 3)

        #for bigram
        bi_first_dict, bi_sec_dict = createKNDict(bi_dict, 2)

        #now find the probability for the trigram sentences
        for quad in quad_dict:
            quad_token = quad.split()
            #W1,W2,W3
            tri_sen = ' '.join(quad_token[:3])

            #the forumula has been divied into parts for easier understanding
            quad_prob1 = max( quad_dict[quad] - d, 0) / tri_dict[tri_sen]
            quad_prob2 = d/tri_dict[tri_sen] * ( quad_sec_dict[tri_sen] )

            tri_prob1 = max( tri_first_dict[quad_token[-1]] - d, 0) / len(tri_dict)
            tri_prob2 = ( d/len(tri_dict) )* ( tri_sec_dict[' '.join(quad_token[1:3])] )

            bi_prob1 = max( bi_first_dict[quad_token[-1]] - d, 0) / len(bi_dict)
            bi_prob2 = ( d/len(bi_dict) ) * ( bi_sec_dict[quad_token[-2]])
            uni_prob = bi_first_dict[quad_token[-1]] / len(bi_dict)

            prob = quad_prob1 + quad_prob2*( tri_prob1 + tri_prob2*( bi_prob1 + bi_prob2* uni_prob ) )

            if tri_sen not in prob_dict:
                prob_dict[tri_sen] = []
                prob_dict[tri_sen].append([prob,quad_token[-1]])
            else:
                prob_dict[tri_sen].append([prob,quad_token[-1]])

def computeKnesserNeyProb1(vocab_dict, bi_dict, tri_dict, quad_dict, prob_dict ):
        d = 0.75

        #first create the dict for storing the count of Wn-1 followed by Wn and for
        #ngrams preceding Wn-1

     ...


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
For Computing the Perplexity
Can you write Python code for it?

#return:float
#arg:list,int,dict,dict,dict,dict
#computes the score for test data
def computePerplexity(test_quadgrams,token_len,tri_dict,quad_dict,vocab_dict,prob_dict):

    perplexity = float(1.0)
    n = token_len

    for item in quad_dict:
        sen_token = item.split()
        sen = ' '.join(sen_token[0:3])
        prob = quad_dict[item]/tri_dict[sen]
        perplexity = perplexity * ( prob**(1./n))
    with open('Test_Scores/Knesser_Ney_Interpolated_Score.txt','a') as w:
        w.write('\nPerplexity: '+str(perplexity))
    return perplexity


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
For Testing the Language Model
Calculates % Accuracy and Perplexity<br>
NOTE : If this is run then no need to run the cells following it
Can you write Python code for it?

train_file = 'training_corpus.txt'
test_file = 'test_corpus.txt'
#load the corpus for the dataset
token_len = trainCorpus(train_file,test_file,bi_dict,tri_dict,quad_dict,vocab_dict,prob_dict)

train_file = 'corpusfile.txt'
token_len = loadCorpus(train_file, bi_dict, tri_dict, quad_dict, vocab_dict)

#compute the Knesser Ney probabilities for bigrams,trigrams and unigrams
computeKnesserNeyProb(vocab_dict, bi_dict, tri_dict, quad_dict, prob_dict )

#sort the probable words by their probability
sortProbWordDict(prob_dict)

#take user input
input_sen = takeInput()

### PREDICTION
start_time2 = time.time()
prediction = doPrediction(input_sen,prob_dict)
print('Word Prediction:',prediction)
print("---Time for Prediction Operation: %s seconds ---" % (time.time() - start_time2))

#FOR DEBUGGING ONLY
writeProbDicts(prob_dict)


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
Interpolated Knesser Ney related Work
For creating Dictionaries required for Knesser Ney Probability Calculation
This function makes dictionaries which keeps count of ngrams where Wn is followed by Wn-1 and also count of ngrams where Wn-1 preceeds the ngram
Can you write Python code for it?

#creates the dictionaries required for computing Interpolated Knesser Ney probability
#return:dict,dict
#arg:dict, int
def createKNDict(ngram_dict, n):
    #for knesser ney probability formula we need to find to important things
    #first is for P(Wn|Wn-1) if find no. of ngrams which ends with Wn and no. of ngrams which starts
    #with Wn-1
    #so we divide the formula into two parts ,first part can be found in constant time
    #and second term is found here
    i = 0
    d = 0.75
    #for storing count of ngram ending with Wn,key:unigram
    first_dict = {}
    #for storing count of ngram having Wn-1 as its starting part, key: trigram sentence
    sec_dict = {}

    for key in ngram_dict:
        #split the key sentence into tokens
        ngram_token = key.split()
        #since the indexing is from 0 ,so for quadgram we need to create a sentence of three words
        #so start from 0 to 2,so we subtract 1,similarly for trigram from 0 to 1
        n_1gram_sen = ' '.join(ngram_token[:n-1])

        #n_1gram_sen is the word that  stars in sec_dict[n_1gram_sen] number of times in ngram_dict
        if n_1gram_sen not in sec_dict:
            sec_dict[ n_1gram_sen ] = 1
        else:
            sec_dict[ n_1gram_sen ] += 1

        if ngram_token[-1] not in first_dict:
            first_dict[ ngram_token[-1] ] = 1
        else:
            first_dict[ ngram_token[-1] ] += 1

    return first_dict, sec_dict


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
Some Misc functions for debugging
Sort the probable words
Can you write Python code for it?

#for sorting the probable word acc. to their probabilities

#returns: void
#arg: dict
def sortProbWordDict(prob_dict):

    for key in prob_dict:
        if len(prob_dict[key])>1:
            #only at most top 2 most probable words have been taken
            prob_dict[key] = sorted(prob_dict[key],reverse = True)[:2]


I am working on the file "Notebooks/Word_Prediction_using_Interpolated_Knesser_Ney.ipynb".
The first task is:
Tokenize and load the corpus data
Can you write Python code for it?

#returns : void
#arg: string,dict,dict,dict,dict
#loads the corpus for the dataset and makes the frequency count of quadgram ,bigram and trigram strings
def loadCorpus(file_path, bi_dict, tri_dict, quad_dict, vocab_dict):

    w1 = ''    #for storing the 3rd last word to be used for next token set
    w2 = ''    #for storing the 2nd last word to be used for next token set
    w3 = ''    #for storing the last word to be used for next token set
    token = []
    #total no. of words in the corpus
    word_len = 0

    #open the corpus file and read it line by line
    with open(file_path,'r') as file:
        for line in file:

            #split the string into word tokens
            temp_l = line.split()
            i = 0
            j = 0

            #does the same as the removePunctuations() function,implicit declratation for performance reasons
            #changes the word to lowercase and removes punctuations from it
            for word in temp_l :
                j = 0
                #print(len(word))
                for l in word :
                    if l in string.punctuation:
                        if l == "'":
                            if j+1<len(word) and word[j+1] == 's':
                                j = j + 1
                                continue
                        word = word.replace(l," ")
                        #print(j,word[j])
                    j += 1

                temp_l[i] = word.lower()
                i=i+1

            #spliting is being done here beacause in sentences line here---so after punctuation removal it should
            #become "here so"
            content = " ".join(temp_l)

            token = content.split()
            word_len = word_len + len(token)

            if not token:
                continue

            #add the last word from previous line
            if w3!= '':
                token.insert(0,w3)

            temp0 = list(ngrams(token,2))

            #since we are reading line by line some combinations of word...


I am working on the file "Notebooks/Word_Prediction_using_Pentagrams_Memory_Efficient.ipynb".
The first task is:
Driver function for doing the prediction
Can you write Python code for it?

#%%timeit
del token[:]
def doPrediction(sen):

    #remove punctuations and make it lowercase
    temp_l = sen.split()
    i = 0

    for word in temp_l :
        for l in word :
            if l in string.punctuation:
                word = word.replace(l," ")
        #token[i] = "".join(l for l in word if l not in string.punctuation)
        #token[i] = word.replace('.','').replace(' ','').replace(',','').replace(':','').replace(';','').replace('!','').replace('?','').replace('(','').replace(')','')
        temp_l[i] = word.lower()
        i=i+1

    content = " ".join(temp_l)
    temp_l = content.split()

    #print(temp_l)
    sen = ' '.join(temp_l)
    #print(sen)

    max_prob = 0
    #when there is no probable word available
    #now for guessing the word which should exist we use quadgram
    right_word = 'apple'

    for word in vocab_dict:
        prob = findprobability(sen,word)
        if prob > max_prob:
            max_prob = prob
            right_word = word

    print('Word Prediction is :',right_word)
    #print('Probability:',max_prob)
    #print(len(token),',',len(vocab))

#print(len(vocab))
sen = input('Enter four words\n')
doPrediction(sen)


I am working on the file "Notebooks/Word_Prediction_using_Quadgrams_Memory_Efficient.ipynb".
The first task is:
Do preprocessing:
Remove the punctuations and lowercase the tokens
Can you write Python code for it?

#returns: string
#arg: string
#remove punctuations and make the string lowercase
def removePunctuations(sen):

      #split the string into word tokens
    temp_l = sen.split()
    i = 0

    #changes the word to lowercase and removes punctuations from it
    for word in temp_l :
        for l in word :
            if l in string.punctuation:
                word = word.replace(l," ")
        temp_l[i] = word.lower()
        i=i+1

    #spliting is being don here beacause in sentences line here---so after punctuation removal it should
    #become "here so"
    content = " ".join(temp_l)

    return content


I am working on the file "Notebooks/Word_Prediction_using_Quadgrams_Memory_Efficient.ipynb".
The first task is:
main function
Can you write Python code for it?

def main():

    #variable declaration
    tri_dict = defaultdict(int)            #for keeping count of sentences of three words
    quad_dict = defaultdict(int)           #for keeping count of sentences of three words
    vocab_dict = defaultdict(int) #for storing the different words with their frequencies

    #load the corpus for the dataset
    loadCorpus('corpusfile.txt',tri_dict,quad_dict,vocab_dict)
    print("---Preprocessing Time: %s seconds ---" % (time.time() - start_time))

    cond = False
    #take input
    while(cond == False):
        sen = input('Enter the string\n')
        sen = removePunctuations(sen)
        temp = sen.split()
        if len(temp) < 3:
              print("Please enter atleast 3 words !")
        else:
              cond = True
              temp = temp[-3:]
              sen = " ".join(temp)

    start_time1 = time.time()
    doPrediction(sen,tri_dict,quad_dict,vocab_dict)
    print("---Time for Prediction Operation: %s seconds ---" % (time.time() - start_time1))

if __name__ == '__main__':
    main()


I am working on the file "Notebooks/Word_Prediction_using_Quadgrams_Memory_Efficient_Encoded_keys.ipynb".
The first task is:
Remove the punctuations and lowercase the tokens
Can you write Python code for it?


I am working on the file "Notebooks/Word_Prediction_using_Quadgrams_Memory_Efficient_Encoded_keys.ipynb".
The first task is:
Tokenize the corpus data
Can you write Python code for it?

#returns : void
#arg: string,dict,dict,dict,list
#loads the corpus for the dataset and makes the frequency count of quadgram and trigram strings
def loadCorupus(filename,tri_dict,quad_dict,vocab_dict,index):
    w1 = ''    #for storing the 3rd last word to be used for next token set
    w2 = ''    #for storing the 2nd last word to be used for next token set
    w3 = ''    #for storing the last word to be used for next token set
    i = 0
    sen = ''
    token = []

    with open(filename,'r') as file:
        #read the data line by line
        for line in file:
            token = line.split()
            i = 0
            for word in token :
                for l in word :
                    if l in string.punctuation:
                        word = word.replace(l," ")
                token[i] = word.lower()
                i=i+1

            content = " ".join(token)
            token = content.split()

            if not token:
                continue

            #first add the previous words
            if w2!= '':
                token.insert(0,w2)
            if w3!= '':
                token.insert(1,w3)

            #tokens for trigrams
            temp1 = list(ngrams(token,3))

            if w1!= '':
                token.insert(0,w1)

            #tokens for quadgrams
            temp2 = list(ngrams(token,4))

            #count the frequency of the trigram sentences
            for t in temp1:
                sen = encodeKey(t,index,vocab_dict)
                tri_dict[sen] += 1

            #count the frequency of the quadgram sentences
            for t in temp2:
                sen = encodeKey(t,index,vocab_dict)
                quad_dict[sen] += 1

            #then take out the last 3 words
            n = len(token)

            w1 = token[n -3]
            w2 = token[n -2]
            w3 = token[n -1]


I am working on the file "Notebooks/Writing Binary Files.ipynb".
The first task is:
Here's the new image:

Now its a checkerboard!
Can you write Python code for it?

a = array([True, False])

(a*255).dtype

#read in the image
cb_img2_read = Image.open('cb_img2.bmp')

#convert to array
checkerboard2_read = np.asarray(cb_img2_read)

#display
matshow(checkerboard2_read)


I am working on the file "Notebooks/archive/Accuracy of generation in NERC by state.ipynb".
The first task is:
Compare 2 data sets
Can you write Python code for it?


I am working on the file "Notebooks/archive/Accuracy of generation in NERC by state.ipynb".
The first task is:
Read EIA-923 data
Can you write Python code for it?

path = os.path.join('Data storage', 'EIA923_Schedules_2_3_4_5_M_12_2015_Final.xlsx')
eia = pd.read_excel(path, header=5, parse_cols='A,G,I,N,O,P,CR')

eia.columns = [col.replace('\n', ' ') for col in eia.columns]
eia.rename(columns={'Net Generation (Megawatthours)': 'Generation (MWh)',
                  'NERC Region': 'region'}, inplace=True)

eia.head()

eia['AER Fuel Type Code'].unique()

fuel_cats = {'Coal': ['COL'],
             'Natural Gas': ['NG'],
             'Hydro': ['HYC', 'HPS'],
             'Nuclear': ['NUC'],
             'Wind': ['WND'],
             'Solar': ['SUN'],
             'Other Renewables': ['GEO', 'ORW', 'WWW', 'MLG'],
             'Other': ['WOO', 'DFO', 'RFO', 'PC', 'OTH', 'OOG', 'WOC']}

for cat, fuels in fuel_cats.iteritems():
    eia.loc[eia['AER Fuel Type Code'].isin(fuels), 'Fuel category'] = cat

eia_nerc = eia.groupby(['region', 'Fuel category']).sum()
eia_nerc.reset_index(inplace=True)

eia_nerc = eia_nerc.pivot_table(index='region', columns=['Fuel category'],
                     values='Generation (MWh)')
eia_nerc.rename(index={'TRE': 'ERCOT'}, inplace=True)
eia_nerc['Total'] = eia_nerc.sum(axis=1)
eia_nerc


I am working on the file "Notebooks/archive/Detrending justification.ipynb".
The first task is:
All index values over time for reference
Can you write Python code for it?

order = ['MRO', 'SPP', 'RFC', 'ERCOT', 'FRCC', 'SERC',
         'WECC', 'NPCC']
FG_kwargs = dict(col='region',
                 hue='region',
                 col_wrap=3,
                 aspect=1.2,
                 hue_order=order,
                 ylim=(0, 1050))

region_facet_grid(df=gen_index, plot_function=plt.plot, x_axis='datetime',
                  y_axis='index (g/kWh)', col_order=order,
                  suptitle='Monthly Index', FG_kwargs=FG_kwargs)


I am working on the file "Notebooks/archive/Detrending justification.ipynb".
The first task is:
An interactive version, in case it's helpful
Can you write Python code for it?

order = ['MRO', 'SPP', 'RFC', 'ERCOT', 'FRCC', 'SERC',
         'WECC', 'NPCC']

trace = []

for region in order:
    df = gen_index.loc[gen_index['region'] == region]
    trace.append(go.Scatter(x=df['datetime'],
                            y=df['index (g/kWh)'],
                            name=region,
                            mode='lines'))

iplot(trace)


I am working on the file "Notebooks/archive/Detrending justification.ipynb".
The first task is:
Look at ERCOT and SPP
These two regions are well-correlated with the original and detrended data.
Without detrending the correlation is 0.92
Can you write Python code for it?

order = ['ERCOT', 'SPP']
trace = []

for region in order:
    df = gen_index.loc[gen_index['region'] == region]
    trace.append(go.Scatter(x=df['datetime'],
                            y=df['index (g/kWh)'],
                            name=region,
                            mode='lines'))

iplot(trace)


I am working on the file "Notebooks/archive/Detrending justification.ipynb".
The first task is:
Look at FRCC and SPP
These two regions are well-correlated with the original data, but uncorrelated once they are detrended. I think it's better to detrend because my question is asking if the regions tend to have high/low carbon intensity at the same times of the year. The correlated downward trend doesn't matter for this analysis.
Without detrending the correlation is 0.84
Can you write Python code for it?

order = ['FRCC', 'SPP']
trace = []

for region in order:
    df = gen_index.loc[gen_index['region'] == region]
    trace.append(go.Scatter(x=df['datetime'],
                            y=df['index (g/kWh)'],
                            name=region,
                            mode='lines'))

iplot(trace)


I am working on the file "Notebooks/archive/Detrending justification.ipynb".
The first task is:
With detrending the correlation is 0.65
Can you write Python code for it?

order = ['ERCOT', 'SPP']
trace = []

for region in order:
    df = gen_index.loc[gen_index['region'] == region]
    trace.append(go.Scatter(x=df['datetime'],
                            y=detrend(df['index (g/kWh)']),
                            name=region,
                            mode='lines'))

iplot(trace)


I am working on the file "Notebooks/archive/Download & clean all 923 data.ipynb".
The first task is:
Assign NERC region to pre-2005/6 facilities based on where they ended up
Somehow I'm having trouble doing this
Can you write Python code for it?

region_dict = dict(nerc_assignment.loc[nerc_assignment['Year'] == 2006,
                                       ['Plant Id', 'NERC Region']].values)

regions = ['ECAR', 'MAPP', 'MAIN', 'MAAC']
years = range(2001, 2006)
nerc_assignment.loc[(nerc_region.isin(regions)) &
                    (nerc_assignment['Year'].isin(years)),
                    'Corrected Region'] = nerc_assignment.loc[(nerc_region.isin(regions)) &
                                                      (nerc_assignment['Year'].isin(years)),
                                                      'Plant Id'].map(region_dict)

nerc_assignment.head()

nerc_assignment.loc[(nerc_assignment['Year'] == 2006) &
                                 (nerc_assignment['Plant Id'] == 3), 'NERC Region'].values[0]

nerc_assignment.loc[(nerc_assignment['Plant Id'] == 3) &
                            (nerc_assignment['Year'].isin(years)), 'Corrected Region'] = 'SERC'

nerc_assignment.loc[(nerc_assignment['Plant Id'] == 3) &
                            (nerc_assignment['Year'].isin(years)), 'Corrected Region']

nerc_assignment.loc[nerc_assignment['Year'] == 2002].head()

nerc_assignment.index = pd.MultiIndex.from_arrays([nerc_assignment['Year'],
                                                   nerc_assignment['Plant Id']])

nerc_assignment.head()

idx = pd.IndexSlice

regions_2006 = nerc_assignment.loc[idx[2006, :], 'NERC Region'].copy()

regions_2006 = nerc_assignment.xs(2006, level='Year')['NERC Region']

regions_2006

for year in range(2001, 2006):
    nerc_assignment.xs(year, level='Year')['Corrected NERC'] = regions_2006

nerc_assignment


I am working on the file "Notebooks/archive/Explore method for allocating state-level data to NERC regions.ipynb".
The first task is:
2015 generation and fuel consumption from annual vs monthly reporting plants
The goal here is to figure out how much of generation and fuel consumption from facilities that only report annually is in each NERC region (by state)
Can you write Python code for it?

path = os.path.join('Data storage', 'EIA923_Schedules_2_3_4_5_M_12_2015_Final.xlsx')
frequency = pd.read_excel(path, sheetname='Page 6 Plant Frame', header=4)

frequency.head()

frequency.rename(columns={'Plant Id': 'plant id',
                          'Plant State': 'state',
                          'YEAR': 'year',
                          'Reporting\nFrequency': 'Reporting Frequency'}, inplace=True)

frequency.head()

frequency.dtypes


I am working on the file "Notebooks/archive/Explore method for allocating state-level data to NERC regions.ipynb".
The first task is:
Filter out data older than 2014 to reduce size
Can you write Python code for it?

facility_df['state'] = facility_df['geography'].str[-2:]
keep_cols = ['fuel', 'year', 'month', 'datetime', 'state', 'plant id', 'NERC',
             'generation (MWh)', 'total fuel (mmbtu)', 'elec fuel (mmbtu)']
facility_df = facility_df.loc[facility_df['year'] >= 2014, keep_cols]

facility_fuel_cats = {'COW': ['SUB', 'BIT', 'LIG', 'WC', 'SC', 'RC', 'SGC'],
                      'NG': ['NG'],
                      'PEL': ['DFO', 'RFO', 'KER', 'JF',
                              'PG', 'WO', 'SGP'],
                      'PC': ['PC'],
                      'HYC': ['WAT'],
                      'HPS': [],
                      'GEO': ['GEO'],
                      'NUC': ['NUC'],
                      'OOG': ['BFG', 'OG', 'LFG'],
                      'OTH': ['OTH', 'MSN', 'MSW', 'PUR', 'TDF', 'WH'],
                      'SUN': ['SUN'],
                      'DPV': [],
                      'WAS': ['OBL', 'OBS', 'OBG', 'MSB', 'SLW'],
                      'WND': ['WND'],
                      'WWW': ['WDL', 'WDS', 'AB', 'BLQ']
                      }

for category in facility_fuel_cats.keys():
    fuels = facility_fuel_cats[category]
    facility_df.loc[facility_df['fuel'].isin(fuels),
                    'fuel category'] = category

facility_df.head()

facility_df.dtypes

facility_df.loc[facility_df['NERC'].isnull(), 'state'].unique()


I am working on the file "Notebooks/archive/Explore method for allocating state-level data to NERC regions.ipynb".
The first task is:
How much generation from large sources (Hydro, wind, coal, natural gas, and nuclear) is missed by monthly 923 data?
Can you write Python code for it?

for fuel in ['HYC', 'WND', 'COW', 'NG', 'NUC', 'SUN']:
    state_total = annual_state.loc[2016, :, fuel]['generation (MWh)'].sum()
    facility_total = annual_facility.loc[2016, :, fuel]['generation (MWh)'].sum()

    error = (state_total - facility_total) / state_total
    print('{} has an error of {:.2f}%'.format(fuel, error * 100))


I am working on the file "Notebooks/archive/Explore method for allocating state-level data to NERC regions.ipynb".
The first task is:
Import facility data and NERC labels
Can you write Python code for it?

path = os.path.join('Data storage', 'Facility gen fuels and CO2 2017-05-25.zip')
facility_df = pd.read_csv(path, parse_dates=['datetime'])

facility_df.head()

facility_df.dropna(inplace=True, subset=['lat', 'lon'])

cols = ['lat', 'lon', 'plant id', 'year']
small_facility = facility_df.loc[:, cols].drop_duplicates()

geometry = [Point(xy) for xy in zip(small_facility.lon, small_facility.lat)]
# small_facility = small_facility.drop(['lon', 'lat'], axis=1)
crs = {'init': 'epsg:4326'}
geo_df = GeoDataFrame(small_facility, crs=crs, geometry=geometry)


I am working on the file "Notebooks/archive/Explore method for allocating state-level data to NERC regions.ipynb".
The first task is:
It's interesting that the facility data has fuel consumption for solar generation and the state data doesn't. Looking at a 923 data file, it's clear that the fuel consumption is just based on a conversion efficiency of 36.6% across all facilities.
Can you write Python code for it?

annual_state.loc[2016, 'CA', 'SUN']

annual_facility.loc[2016, 'CA', 'SUN']


I am working on the file "Notebooks/archive/Explore method for allocating state-level data to NERC regions.ipynb".
The first task is:
Make a dataframe with generation, fuel consumption, and reporting frequency of facilities in 2015
Can you write Python code for it?

freq_cols = ['year', 'plant id', 'Reporting Frequency']
df = pd.merge(facility_df, frequency.loc[:, freq_cols], on=['year', 'plant id'])

df.head()

g = sns.factorplot(x='fuel category', y='generation (MWh)', hue='Reporting Frequency',
                   col='NERC', col_wrap=3, data=df, estimator=np.sum, ci=0, kind='bar',
                   palette='tab10')

g.set_xticklabels(rotation=30)


I am working on the file "Notebooks/archive/Explore method for allocating state-level data to NERC regions.ipynb".
The first task is:
Merge NERC labels into the facility df
Can you write Python code for it?

cols = ['plant id', 'year', 'NERC']
facility_df = facility_df.merge(facility_nerc.loc[:, cols],
                                on=['plant id', 'year'], how='left')

facility_df.head()


I am working on the file "Notebooks/archive/Explore method for allocating state-level data to NERC regions.ipynb".
The first task is:
Read NERC shapefile and merge with geo_df
Can you write Python code for it?

path = os.path.join('Data storage', 'NERC_Regions_EIA', 'NercRegions_201610.shp')
regions = gpd.read_file(path)

facility_nerc = gpd.sjoin(geo_df, regions, how='inner', op='within')

facility_nerc.head()


I am working on the file "Notebooks/archive/Explore method for allocating state-level data to NERC regions.ipynb".
The first task is:
States that include more than one NERC region
Can you write Python code for it?

NERC_states = ['WY', 'SD', 'NE', 'OK', 'TX', 'NM', 'LA', 'AR',
               'MO', 'MN', 'IL', 'KY', 'VA', 'FL']

error_list = []
for state in NERC_states:
    error = (annual_state.loc[2016, state]
             - annual_facility.loc[2016, state]) / annual_state.loc[2016, state]
    error['state'] = state

    for col in ['generation (MWh)']:#, 'elec fuel (mmbtu)']:
        if error.loc[error[col] > 0.05, col].any():
            error_list.append(error.loc[error[col] > 0.05])


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Because I have monthly data for every facility from 2001-2017, there are lots of duplicate rows. No need to do a spatial join on every row. Just keep one instance of each facility in each year.
Can you write Python code for it?

cols = ['lat', 'lon', 'plant id', 'year']
small_facility = facility_df.loc[:, cols].drop_duplicates()


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Drop plants without lat/lon data
From 2001-mid 2017, 9 facilities (out of 8,435) don't have valid lat/lon data. All of them are from 2010 or earlier. Drop these rows or the spatial join fails. The amount of generation from these plants is tiny - well under 0.05% in all years.
Can you write Python code for it?

print(len(facility_df['plant id'].unique()), 'total plants')
print(len(facility_df.loc[facility_df['lat'].isnull(), 'plant id'].unique()),
      'plants without lat/lon')

years = facility_df.loc[facility_df['lat'].isnull(), 'year'].unique()
for year in years:
    total_gen = facility_df.loc[facility_df['year'] == year, 'generation (MWh)'].sum()

    # Plant ids with no 'lat' in year
    no_loc_plants = facility_df.loc[(facility_df['lat'].isnull()) &
                                    (facility_df['year'] == year), 'plant id'].unique()

    no_loc_gen = facility_df.loc[(facility_df['year'] == year) &
                                 (facility_df['plant id'].isin(no_loc_plants)),
                                  'generation (MWh)'].sum()

    percent_dropped = no_loc_gen / total_gen * 100

    print('In {}, {:.3f}% of generation is from plants without lat/lon'.format(year, percent_dropped))

facility_df.dropna(inplace=True, subset=['lat', 'lon'])

facility_df.columns


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
I figured out that plant id 641 is one that shows up in both FRCC and SERC
Can you write Python code for it?

plant_641 = geo_df.loc[geo_df['plant id'] == 641, 'geometry'].values[0]

frcc.contains(plant_641)

serc.contains(plant_641)


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Method 1 - use defaults
Can you write Python code for it?

df1 = gpd.sjoin(regions, geo_df)

df1.head()


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Method 2 (faster when using the default operation)
Can you write Python code for it?

facility_nerc = gpd.sjoin(df, geo_df, how="inner")

facility_nerc.head()


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Method 2 - use 'within'
Can you write Python code for it?

df2 = gpd.sjoin(geo_df, regions, op='within')

df2.head()


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Method 3 - use 'contains'
Can you write Python code for it?

df3 = gpd.sjoin(regions, geo_df, op='contains')

df3.head()


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
NEW - read file with only plant id, state, and lat/lon
Link to data file
If you need to load the shapefile, download it here.
Can you write Python code for it?

# If loading the shapefile
shape_path = ''
regions = gpd.read_file(shape_path)

path = join(data_path, 'Facility labels', 'Facility locations.csv')
location = pd.read_csv(path)

len(location)


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
No difference in score when X is preprocessed with StandardScaler
Can you write Python code for it?

clf.score(X_scale, y)

clf = neighbors.KNeighborsClassifier(n_neighbors, weights='uniform')
clf.fit(X_train, y_train)

clf.score(X_test, y_test)

clf = neighbors.KNeighborsClassifier(n_neighbors, weights='distance')
clf.fit(X_train, y_train)

clf.score(X_test, y_test)

h = .02  # step size in the mesh

# Create color maps
cmap_light = ListedColormap(['#FFAAAA', '#AAFFAA', '#AAAAFF'])
cmap_bold = ListedColormap(['#FF0000', '#00FF00', '#0000FF'])

for weights in ['uniform', 'distance']:
    # we create an instance of Neighbours Classifier and fit the data.
    clf = neighbors.KNeighborsClassifier(n_neighbors, weights=weights)
    clf.fit(X, y)

    # Plot the decision boundary. For that, we will assign a color to each
    # point in the mesh [x_min, x_max]x[y_min, y_max].
    x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                         np.arange(y_min, y_max, h))
    Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

    # Put the result into a color plot
    Z = Z.reshape(xx.shape)
    plt.figure()
    plt.pcolormesh(xx, yy, Z, cmap=cmap_light)

    # Plot also the training points
    plt.scatter(X[:, 0], X[:, 1], c=y, cmap=cmap_bold,
                edgecolor='k', s=20)
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())
    plt.title("3-Class classification (k = %i, weights = '%s')"
              % (n_neighbors, weights))


I am working on the file "Notebooks/archive/Trying geopandas to assign NERC regions.ipynb".
The first task is:
Now using NERC region shapefiles created by DHS
Can you write Python code for it?

path = os.path.join(data_path, 'NERC_Regions', 'NERC_Regions.shp')
regions = gpd.read_file(path)
regions.columns = regions.columns.str.lower()
regions['nerc'] = regions['name'].str.split().str[-1].str.strip('()')
# regions = regions.loc[:, ['nerc', 'geometry']]
regions.head()

regions.loc[regions['nerc']=='WECC'].plot(cmap='tab20', alpha=0.5)

regions.plot()

regions.to_crs(epsg=2163).plot(cmap='tab20', alpha=0.5,
                               linewidth=0.5, edgecolor='0.1')

# This is slow!

regions_nerc = regions.dissolve(by='nerc', as_index=False)

regions_nerc.to_crs(epsg=2163).plot()

path = join(data_path, 'NERC_Regions', 'Aggregated NERC Regions.geojson')
regions_nerc.to_file(path, driver='GeoJSON')

fiona.open()

path = join(data_path, 'NERC_Regions', 'Aggregated NERC Regions.geojson')
regions_nerc = gpd.read_file(path, driver='GeoJSON')

path = os.path.join(data_path, 'cb_2016_us_state_20m', 'cb_2016_us_state_20m.shp')
states = gpd.read_file(path)
states.crs

drop_states = ['Alaska', 'Hawaii', 'Puerto Rico']
states = states.loc[~states['NAME'].isin(drop_states)]

states.to_crs(epsg=2163).plot()

path = join(data_path, 'final NERC data',
            'Summary table {}.csv'.format(file_date))
index = pd.read_csv(path)
index

for nerc in regions_nerc['nerc'].unique():
    try:
        val_2017 = index.loc[index['nerc']==nerc, '2017'].values[0]
        val_2001 = index.loc[index['nerc']==nerc, '2001'].values[0]
        reduce = index.loc[index['nerc']==nerc, 'Percent Reduction'].values[0]
#         print(val)
        regions_nerc.loc[regions_nerc['nerc']==nerc, 2017] = val_2017
        regions_nerc.loc[regions_nerc['nerc']==nerc, 2001] = val_2001

        regions_nerc.loc[regions_nerc['nerc']==nerc, 'reduction'] = '{:.0%}'.format(reduce)
        regions_nerc.loc[regions_nerc['nerc']==nerc, 'reduction value'] = reduce
    except:
        pass

regions_nerc

regions_albers = regions_nerc.to_crs(epsg=2163)
states_albers = states.to_crs(epsg=2163)

regions_albers.plot(column=2001, cmap='cividis_r',...