I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
We also need  to get out the most common words by their document frequency for each model. The function below, top_vocab_by_docfreq will do this each time we loop and create a new model:
Can you write Python code for it?

def top_vocab_by_docfreq(df, num_words):
    '''
    Retrieve the most common words (by document frequency) for a given model
    '''
    docfreq_df = df > 0
    wordcolumn_sums = docfreq_df.sum()
    words_by_freq = wordcolumn_sums.sort_values(ascending=False)
    top_words = words_by_freq[:num_words]
    top_words_list = top_words.index.tolist()

    return top_words_list


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
We can use sklearn's DictVectorizer to turn this into a DTM:
Can you write Python code for it?

from sklearn.feature_extraction import DictVectorizer

dv = DictVectorizer()
dtm = dv.fit_transform(freqdict_list)
term_list = dv.feature_names_

dtm


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
We don't have 100 minutes!
Efficiency
A lot of that time is figuring out the vocabulary:
Can you write Python code for it?


I am working on the file "08-Classification/02-Underwood-Sellers.ipynb".
The first task is:
We then need to normalize the frequencies:
Can you write Python code for it?

def normalize_model(train_df_, test_df_, vocabulary):
    '''
    Normalize the model's term frequencies and put them into standard units
    '''
    # Select columns for only the most common words
    train_df_ = train_df_[vocabulary]
    test_df_ = test_df_[vocabulary]

    # Normalize each value by the sum of all values in its row
    train_df_ = train_df_.apply(lambda x: x/sum(x), axis=1)
    test_df_ = test_df_.apply(lambda x: x/sum(x), axis=1)

    # Get mean and stdev for each column
    train_mean = np.mean(train_df_)
    train_std = np.std(train_df_)

    # Transform each value to standard units for its column
    train_df_ = ( train_df_ - train_mean ) / train_std
    test_df_ = ( test_df_ - train_mean ) / train_std

    return train_df_, test_df_


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
But dictionaries are not orderable: we need a way to tell the function how to sort our data.
We can do this by specifying the key function, a function which given an item returns the sorting key for that item:
Can you write Python code for it?

# sort alphabetically by first name
sorted(data, key=lambda item: item['first'])

# sort by year of birth
sorted(data, key=lambda item: item['YOB'])


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
But now we can use the function to explore new things, such as the effect of new starting values:
Can you write Python code for it?


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
Default Argument Values
Often when defining a function, there are certain values that we want the function to use most of the time, but we'd also like to give the user some flexibility.
In this case, we can use default values for arguments.
Consider the fibonacci function from before.
What if we would like the user to be able to play with the starting values?
We could do that as follows:
Can you write Python code for it?

def fibonacci(N, a=0, b=1):
    L = []
    while len(L) < N:
        a, b = b, a + b
        L.append(a)
    return L


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
Here it is not the names args and kwargs that are important, but the * characters preceding them.
args and kwargs are just the variable names often used by convention, short for "arguments" and "keyword arguments".
The operative difference is the asterisk characters: a single * before a variable means "expand this as a sequence", while a double ** before a variable means "expand this as a dictionary".
In fact, this syntax can be used not only with the function definition, but with the function call as well!
Can you write Python code for it?

inputs = (1, 2, 3)
keywords = {'pi': 3.14}

catch_all(*inputs, **keywords)


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
If you're familiar with strongly-typed languages like C, you'll immediately notice that there is no type information associated with the function inputs or outputs.
Python functions can return any Python object, simple or compound, which means constructs that may be difficult in other languages are straightforward in Python.
For example, multiple return values are simply put in a tuple, which is indicated by commas:
Can you write Python code for it?

def real_imag_conj(val):
    return val.real, val.imag, val.conjugate()

r, i, c = real_imag_conj(3 + 4j)
print(r, i, c)


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
Now suppose we want to sort this data.
Python has a sorted function that does this:
Can you write Python code for it?


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
Now we have a function named fibonacci which takes a single argument N, does something with this argument, and returns a value; in this case, a list of the first N Fibonacci numbers:
Can you write Python code for it?


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
So why would you ever want to use such a thing?
Primarily, it comes down to the fact that everything is an object in Python, even functions themselves!
That means that functions can be passed as arguments to functions.
As an example of this, suppose we have some data stored in a list of dictionaries:
Can you write Python code for it?

data = [{'first':'Guido', 'last':'Van Rossum', 'YOB':1956},
        {'first':'Grace', 'last':'Hopper',     'YOB':1906},
        {'first':'Alan',  'last':'Turing',     'YOB':1912}]


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
The values can also be specified by name if desired, in which case the order of the named values does not matter:
Can you write Python code for it?


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
This lambda function is roughly equivalent to
Can you write Python code for it?


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
When non-keyword arguments are used together with keyword arguments, the keyword arguments must come at the end.
Defining Functions
Functions become even more useful when we begin to define our own, organizing functionality to be used in multiple places.
In Python, functions are defined with the def statement.
For example, we can encapsulate a version of our Fibonacci sequence code from the previous section as follows:
Can you write Python code for it?

def fibonacci(N):
    L = []
    a, b = 0, 1
    while len(L) < N:
        a, b = b, a + b
        L.append(a)
    return L


I am working on the file "08-Defining-Functions.ipynb".
The first task is:
With a single argument, the result of the function call is identical to before:
Can you write Python code for it?


I am working on the file "08-LemCorpCrit.ipynb".
The first task is:
Decreasing
This is the same process here : we want to see which terms tend to <b>disassociate</b> themselves from "littérature" (by default, which you can change with the trained models). Then again, you have to check the p values. "transplantation", "choeur" and "philé" are not considered significant, "chaldéen" is, and "destination", "morceau", etc. are as well. The fact that those are less significant is logical : the fewer the terms, the more erratic their series tend to be.
Can you write Python code for it?

df1.sort_values('slope', ascending=True).head(50)

for i, row in df1.sort_values('slope', ascending=True).head(20).iterrows():
    plot_cosine_series('littérature', row['token'], 3, 2)

def intersect_neighbor_vocab(anchor, topn=2000):

    vocabs = []

    for year, model in models.items():
        similar = model.most_similar(anchor, topn=topn)
        vocabs.append(set([s[0] for s in similar]))

    return set.intersection(*vocabs)

intersect_vocab = intersect_neighbor_vocab('littérature')

data = []
for token in intersect_vocab:

    series = cosine_series('littérature', token)
    fit = lin_reg(series)

    if fit.pvalues[1] < 0.05:
        data.append((token, fit.params[1], fit.pvalues[1]))

import pandas as pd

df2 = pd.DataFrame(data, columns=('token', 'slope', 'p'))


I am working on the file "08-LemCorpCrit.ipynb".
The first task is:
In this following cell, we show how the top ten most similar vectors change through time compared to "littérature" : "humanisme" for example seems to be very rare before 1860, and then becomes more and more similar to "littérature". Those show the terms that were not similar in the beginning, but <b>tend to be more and more related</b> to "littérature". You should keep in mind the p values associated to each vector.
Can you write Python code for it?

for i, row in df1.sort_values('slope', ascending=False).head(20).iterrows():
    plot_cosine_series('littérature', row['token'], 3, 2)


I am working on the file "08-LemCorpCrit.ipynb".
The first task is:
Increasing
In this part, we want to show what terms emerge more and more with the main entry, that is to say by default "littérature". The "slope" is the degree of progress, and the "p" value its efficiency. So here, the main emergence with "littérature" which is significant is "humanisme". All terms seem to be significant, except "fédéralisme", "welschinger", "maniere", "hennet", "réapparition", "deffence", "bourgin", "colonie", "naturalisme", "réalisme", "sillery", "gréco", "compétence", "symbolisme", "catholique", "japonais", "manuel", "romand", "topographie, "organisme", "prédominance". That is to say that those terms can be nearest, but that statistically they are not significant enough to be sure, while the others are more certain.
Can you write Python code for it?

pd.set_option('display.max_rows', 1000)

df1.sort_values('slope', ascending=False).head(50)


I am working on the file "08-LemCorpCrit.ipynb".
The first task is:
Intersected neighbors
In this part, we show which significant terms tend to be, throughout time, the nearest neighbours to the main entry. That is to say that these vectors follow the same evolution through time as the main entry and are very near to the "littérature" vector. At this stage, we only ask for significant terms (filter above : "if fit.pvalues[1] < 0.05").
Can you write Python code for it?

df2.sort_values('slope', ascending=False)

for i, row in df2.sort_values('slope', ascending=False).iterrows():
    plot_cosine_series('littérature', row['token'], 3, 2)


I am working on the file "08-LemCorpCrit.ipynb".
The first task is:
The two next cells get the 200 most similar terms to a specific term, from the training models, here "littérature".
Can you write Python code for it?

def union_neighbor_vocab(anchor, topn=200):

    vocab = set()

    for year, model in models.items():
        similar = model.most_similar(anchor, topn=topn)
        vocab.update([s[0] for s in similar])

    return vocab

union_vocab = union_neighbor_vocab('littérature')


I am working on the file "08-Machine-Learning-I/HW08/README.ipynb".
The first task is:
Question 4
Write a function called "rating_confusion" which takes one argument:

cer, which is a pandas data frame

and returns:

cf, a confusion matrix where the rows correspond to predicted_ratingID and the columns correspond to ratingID.
Can you write Python code for it?

#### play with code here #####

# Hint: look up pandas "crosstab"


I am working on the file "08-Machine-Learning-I/HW08/README.ipynb".
The first task is:
Sample output:
In [1]: code_shift(df[:5])
Out[1]:     offense_id  Shift   ShiftID
        0   172490115   Morn    1
        1   172490265   Eve     3
        2   172490322   Morn    1
        3   172490390   Morn    1
        4   172490401   Morn    1
For the last 2 questions, you will use the cereal data file available from https://perso.telecom-paristech.fr/eagan/class/igr204/datasets. Execute the download and loading instructions below.
Can you write Python code for it?

%%sh
## RUN BUT DO NOT EDIT THIS CELL
## run this cell to download the cereal dataset into your current directory
wget https://perso.telecom-paristech.fr/eagan/class/igr204/datasets/cereal.csv

## RUN BUT DO NOT EDIT THIS CELL

# load the data, define ratingID
cer = pd.read_csv('cereal.csv', skiprows=[1], delimiter=';')
cer['ratingID'] = cer['rating'].apply(lambda x: 0 if x<60 else 1)

# define predicted ratingID
np.random.seed(12345)
cer['predicted_ratingID'] = (cer['rating']+20*np.random.randn(len(cer))).apply(lambda x: 0 if x<60 else 1)


I am working on the file "08-Machine-Learning-I/HW08/README.ipynb".
The first task is:
Sample output:
In [1]: variable_helper(df[['offense_id','beat','x','y']])
Out[1]: {'beat': 'categorical',
         'offense_id': 'ordinal',
         'x': 'numeric',
         'y': 'numeric'}
Short explanation: offense_id is a number assigned to each offense. There is a natural ordering implied in the id number (based on order of occurrence). Because of this, offense_id is an ordinal feature. The beat uses a numeric label, but refers to a geographic location. There is no natural ordering, so beat is a categorical feature. The location variables (x and y) are numeric position coordinates.
Question 2
Write a function called "get_categories" which takes one argument:

df, which is a pandas data frame

and returns:

cat, a dictionary where keys are names of columns of df corresponding to categorical features, and values are arrays of all the unique values that the feature can take.
Can you write Python code for it?


I am working on the file "08-ModtranFileProcessing.ipynb".
The first task is:
IEMSCT = 1: this example loads data from all columns with data in the sample file.
Can you write Python code for it?

# to load a tape7 file (only selected columns loaded).
tape7 = rymodtran.loadtape7("tape7-02", ['FREQ', 'TOT_TRANS', 'PTH_THRML',
                                         'THRML_SCT', 'SURF_EMIS',
                                         'GRND_RFLT', 'TOTAL_RAD', 'DEPTH',
                                         'DIR_EM', 'BBODY_T[K]'] )
print(tape7.shape)


I am working on the file "08-ModtranFileProcessing.ipynb".
The first task is:
Range scaling atmospheric transmittance and path radiance
If the spectral transmittance at one range is known, it can be scaled to other distances by applying Beer/Bouguer's law.
Calculate the attenuation coefficient for the known transmittance and then apply it to other distances.
$$
\tau_a(R_1) = \exp(-R_1 \alpha_a)
$$
$$\alpha_a = -\log(\tau_a(R_1))/R_1
$$
and then calculate the transmittance at the required range by
$$
\tau_a(R) = \exp(-R \alpha_a)
$$
Path radiance can be scaled by multiplication/division by $(1-\tau)$. Divide the known path radiance by  $(1-\tau_\textrm{known})$ and then multiply with $(1-\tau_\textrm{new})$. The path radiance for an infinitely long path can be calculated and then rescaled:
$$
L_\textrm{path}(R\rightarrow \infty)
=
\frac{L_\textrm{path1}}{1-\tau_\textrm{path1}}
=
\frac{L_\textrm{path2}}{1-\tau_\textrm{path2}}
$$
\ac{MODTRAN} was used to calculate path radiance and transmittance for several distances, using the same atmospheric model. The path is horizontal, at sea level. The Tropical climate model is used with a Rural aerosol and 23 km visibility.
Can you write Python code for it?

# to demonstrate recaling transmittance from one range to another.
pranges = [100,1000,10000]

p = ryplot.Plotter(1,2,2,figsize=(16,8))
for prange in pranges:
    colsPath = ['FREQ', 'TOT_TRANS', 'TOTAL_RAD','DEPTH']
    tape7 = rymodtran.loadtape7(f"data/path{prange}/tape7", colsPath)
    dfPath = pd.DataFrame(tape7,columns=colsPath)
    dfPath = dfPath.rename(columns={'FREQ': 'nu','TOT_TRANS':'tauPath','TOTAL_RAD':'radPathNu'})
    wlp, dfPath['radPathWl'] = ryutils.convertSpectralDensity(dfPath['nu'].values,dfPath['radPathNu'].values,'nl')
    dfPath['radPathWl'] *= 1.0e4
    dfPath['tauPath'] += 1e-6 # prevent zero transmittance

    dfPath['wl'] = 1e4 / dfPath['nu']
#     dfPath['alphaP'] = -np.log(dfPath['tauPath']) / prange
    dfPath['alphaP'] = dfPath['DEPTH'] / prange

    dfPath['radPathWlNorm'] = dfPath['radPathWl'] / (1 - dfPath['tauPath'] )
#     print(dfPath[dfPath['tauPath']<0.01])
    p.plot(1,dfPath['wl'],dfPath['radPathWl'],'Path radiance over distance','Wavelength [$\mu$m]','Radiance W/(m2.sr.um)',
           xAxisFmt='%.2f',label=[f'{prange} m'])
    p.plot(2,dfPath['wl'],dfPath['tauPath'],'Path transmittance over distance','Wavelength [$\mu$m]','Transmittance',
           xAxisFmt='%.2f',label=[f'{prange} m'])
    p.plot(3,dfPath['wl'],dfPath['radPathWlNorm'],'Path radiance / (1-$\\tau$)','Wavelength [$\mu$m]','Radiance W/(m2.sr.um)',
           xAxisFmt='%.2f',label=[f'{prange} m'])
    p.semilogY(4,dfPath['wl'],dfPath['alphaP'],'Path attentuation coefficient','Wavelength [$\mu$m]','Attenuation [1/m]',
           xAxisFmt='%.2f',label=[f'{prange} m'])


I am working on the file "08-ModtranFileProcessing.ipynb".
The first task is:
Repeat the previous example, but this time plot and integrate in the wavelength domain. The integrated results should be the same as for the example above.  In this example the wavenumber domain is converted to wavelength domain and the spectral densities are likewise converted.
Note that the domain and densities are simply converted, not resampled. This means that the data is still at constant wavenumber intervals (varying wavelength intervals). This does not pose a problem when plotting, but care must be taken when integrating to tell trapz that the x interval is not constant. Note, however that the data is in a sequence of increasing wavenumber; that means decreasing wavelength, and as a result the integral is negative (negative integration intervals).
Can you write Python code for it?

# to plot different contributions to total path radiance on wavelength scale.
colSelect =  ['FREQ', 'PTH_THRML','PTH_THRML','SOL_SCAT','SING_SCAT', 'TOTAL_RAD']
skyrad= rymodtran.loadtape7("NIRscat.fl7", colSelect )
sr = ryplot.Plotter(1, 5,1,"Path Radiance in NIR, Path to Space from 3 km",
                    figsize=(12,10))

# plot the components separately
for i in [1,2,3,4,5]:
    # convert spectral density from wavenumber units to wavelength
    wl, Lpath = ryutils.convertSpectralDensity(skyrad[:,0],skyrad[:,i],'nl')
    # convert cm2 to m2 for area
    Lpath *= 1.0e4
    sr.plot(i,  wl, Lpath, "","Wavelength [$\mu$m]",
          "L [W/(m$^2$.sr.$\mu$m)]",
         label=[colSelect[i][:]],legendAlpha=0.5, #pltaxis=[0.4,1, 0, 1],
         maxNX=10, maxNY=4, powerLimits = [-4,  4, -5, 5],xAxisFmt='%.2f');

    #convert from /cm^2 to /m2 and integrate using the wavenumber vector
    totinband = - np.trapz(Lpath.reshape(-1, 1),wl,
                                  axis=0)[0]
    print('{0} integral is {1} [W/(m^2.sr)]'.format(colSelect[i][:],totinband))

print('Note that multiple scatter contributes significantly'
      ' to total path radiance')


I am working on the file "08-ModtranFileProcessing.ipynb".
The first task is:
Tape7 Units
The tape7 format, and tools to read the files are given in 08-ModtranFileProcessing.ipynb.  Please consult the notebook for more details.  Here is a summary of the units used  in  tape7 files.
The format differs depending on the type of \ac{MODTRAN} run.
\ac{MODTRAN} does calculations in the wavenumer domain with units \si{\centi\metre^{-1}}.
In most (all?) tape7 files the wavenumber is the first column in the file.  The column header is FREQ.
Depending on the specific tape5 parameters, the transmittance and optical depth may be in different columns, often with different headers.  Column headers can be TOT TRANS or COMBIN_TRANS. The file formats are described below.
Path radiance is only present in some file formats.  Radiance units in tape7 files are \si{\watt\per(\centi\metre\squared.\steradian.\centi\metre^{-1})}.
The first part of the denominator represents areal density in \si{\centi\metre\squared}, hence the density must be converted to \si{\metre\squared}.
The second part of the denominator represents solid angle and no further conversion is necessary.
The third part of the denominator represents the spectral density \si{\per\centi\metre^{-1}} and this can be used as-is or it can be converted to a spectral density in the wavelength domain with units \si{\watt\per(\centi\metre\squared.\steradian.\micro\metre)}.
Recall the relationship between wavelength, $\lambda$, and wavenumber, $\tilde{\nu}$:
$\lambda=10^4/\tilde{\nu}$,  and for spectral density
$dL_\lambda
=dL_{\tilde{\nu}} 10^4/\lambda^2
=dL_{\tilde{\nu}} \tilde{\nu}^2/10^4
$
Loading Tape7 Files
This function reads in the tape7 file from \ac{MODTRAN}  \cite{modtran2009} code, that is used to model the propagation of the electromagnetic radiation through the atmosphere. tape7 is a primary file that contains all the spectral results of the \ac{MODTRAN} run. The header information in the tape7 file contains portions of the tape5 information that is ignored here. The header section in tape7 is followed by a list of spectral samples with...
Can you write Python code for it?

# to demonstrate downloading files.
# get_ipython().system('dir')
tgzFilename = 'modtrandata.tgz'
destinationDir = '.'
tarFilename = 'modtrandata.tar'
url = 'https://raw.githubusercontent.com/NelisW/pyradi/master/pyradi/data/'
dlNames = ryfiles.downloadUntar(tgzFilename, url, destinationDir, tarFilename)

print(type(dlNames))
print(f'filesAvailable are (only a subset printed here):\n{dlNames[:10]}')


I am working on the file "08-ModtranFileProcessing.ipynb".
The first task is:
The final example reads path radiance data from a file, plots the results, and calculates the total path radiance in a spectral band.  In this example the plot and integration are done in wavenumber domain. Normally you would multiply with a sensor spectral response before integration, but this calculation is over the whole band, equally weighted.
The \ac{MODTRAN} path radiance must be converted to metre-based units, see the introduction above.
Can you write Python code for it?

# to plot different contributions to total path radiance on wavenumber scale.

colSelect =  ['FREQ', 'PTH_THRML','THRML_SCT','SOL_SCAT','SING_SCAT', 'TOTAL_RAD']
skyrad= rymodtran.loadtape7("NIRscat.fl7", colSelect )
sr = ryplot.Plotter(1, 5,1,"Path Radiance in NIR, Path to Space from 3 km",
                    figsize=(12,10))
# plot the components separately
for i in [1,2,3,4,5]:
    Lpath = 1.0e4 * skyrad[:,i]
    sr.plot(i,  skyrad[:,0], Lpath, "","Wavenumber [cm$^{-1}$]",
          "L [W/(m$^2$.sr.cm$^{-1}$)]",
         label=[colSelect[i][:]],legendAlpha=0.5, #pltaxis=[0.4,1, 0, 1],
         maxNX=10, maxNY=4, powerLimits = [-4,  4, -5, 5],yAxisFmt='%.2e');

    #convert from /cm^2 to /m2 and integrate using the wavenumber vector
    totinband = np.trapz(Lpath.reshape(-1, 1),skyrad[:,0],
                                  axis=0)[0]
    print('{0} integral is {1} [W/(m^2.sr)]'.format(colSelect[i][:],totinband))

print('Note that multiple scatter contributes significantly' +       ' to total path radiance')


I am working on the file "08-ModtranFileProcessing.ipynb".
The first task is:
The function rymodtran.loadtape7 is relatively easy to use, with only two parameters: the filename and the specification of which columns to read (a list). In some instances (IEMSCT == 0) the column header is given over two lines, and your specification must include the information on both lines. This is done by concatenating the two parts with an underscore character as in TRACE and TRANS becoming TRACE_TRANS.
IEMSCT = 0: this example loads data from five columns. Note the underscore to indicate that the column header flows over two lines.
Can you write Python code for it?

# to load a tape7 file (only selected columns loaded).
tape7 = rymodtran.loadtape7("tape7-01", ['FREQ_CM-1', 'COMBIN_TRANS',
                                         'MOLEC_SCAT', 'AER+CLD_TRANS',
                                         'AER+CLD_abTRNS'] )
print(tape7.shape)


I am working on the file "08-ModtranFileProcessing.ipynb".
The first task is:
The next example plots the transmittance from the individual molecules and aerosol separately to clearly show the effect of each.
Can you write Python code for it?

# to plot the individual transmittance components.
colSelect =  ['FREQ_CM-1', 'COMBIN_TRANS', 'MOLEC_SCAT', 'CO2_TRANS',
              'H2O_TRANS', 'H2O_CONT', 'CH4_TRANS', 'O3_TRANS', 'O2_TRANS',
              'N2O_TRANS', 'AER+CLD_TRANS', 'SO2_TRANS']
tape7= rymodtran.loadtape7("horizon5kmtropical.fl7", colSelect )
wavelen = ryutils.convertSpectralDomain(tape7[:,0],  type='nl')
mT = ryplot.Plotter(1, 9, 1,"Modtran Tropical, 23 km Visibility (Rural)"                   + ", 5 km Path Length",figsize=(12,21))
mT.semilogX(1, wavelen, tape7[:,1], '','', '', maxNY=4,
       label=colSelect[1:],legendAlpha=0.5, pltaxis=[0.2,15, 0, 1]);
mT.semilogX(2, wavelen, tape7[:,2], '','', '', maxNY=4,
       label=colSelect[2:],legendAlpha=0.5, pltaxis=[0.2,15, 0, 1]);
mT.semilogX(3, wavelen, tape7[:,10], '','', '', maxNY=4,
       label=colSelect[10:],legendAlpha=0.5, pltaxis=[0.2,15, 0, 1]);
mT.semilogX(4, wavelen, tape7[:,4] , '','', '', maxNY=4,
       label=colSelect[4:],legendAlpha=0.5, pltaxis=[0.2,15, 0, 1]);
mT.semilogX(5, wavelen, tape7[:,5] , '','', '', maxNY=4,
       label=colSelect[5:],legendAlpha=0.5, pltaxis=[0.2,15, 0, 1]);
mT.semilogX(6, wavelen, tape7[:,3]  , '','', '', maxNY=4,
       label=colSelect[3:],legendAlpha=0.5, pltaxis=[0.2,15, 0, 1]);
mT.semilogX(7, wavelen, tape7[:,6]  , '','', '', maxNY=4,
       label=colSelect[6:],legendAlpha=0.5, pltaxis=[0.2,15, 0, 1]);
mT.semilogX(8, wavelen, tape7[:,7] * tape7[:,8] , '','', '', maxNY=4,
       label=colSelect[7:],legendAlpha=0.5, pltaxis=[0.2,15, 0, 1]);
mT.semilogX(9, wavelen, tape7[:,9]  , '','', '', maxNY=4,
       label=colSelect[9:],legendAlpha=0.5, pltaxis=[0.2,15, 0, 1]);

# to plot an annotated graph showing band definitions.
import matplotlib.patches as patches
colSelect =  ['FREQ_CM-1', 'COMBIN_TRANS']
tape7= rymodtran.loadtape7("horizon5kmtropical.fl7", colSelect )
wavelen = ryutils.convertSpectralDomain(tape7[:,0],  type='nl')
mT = ryplot.Plotter(1, 1, 1,"Tropical: 27 $^\circ$C, 75% RH, 23 km Visibility (Rural)"
                    + ", 5...


I am working on the file "08-ModtranFileProcessing.ipynb".
The first task is:
The next example reads data from the file and display the data in a plot. The columns to be read and plotted are defined in the list colSelect. This list is used when loading the data, but also as legend in the plot.  The \ac{MODTRAN} data is given in wavenumber domain. The plot must be in wavelength domain, hence wavenumber is converted to wavelength using ryutils.convertSpectralDomain. Finally, the various transmittance curves are plotted against wavelength.
Can you write Python code for it?

# to plot different contributions to total transmittance.
colSelect =  ['FREQ_CM-1', 'COMBIN_TRANS', 'H2O_TRANS', 'UMIX_TRANS',
              'O3_TRANS', 'H2O_CONT', 'MOLEC_SCAT', 'AER+CLD_TRANS']
tape7= rymodtran.loadtape7("tape7VISNIR5kmTrop23Vis", colSelect )
wavelen = ryutils.convertSpectralDomain(tape7[:,0],  type='nl')
mT = ryplot.Plotter(1, 1, 1,"Modtran Tropical, 23 km Visibility (Rural)"
                   + ", 5 km Path Length",figsize=(12,6))
mT.plot(1, wavelen, tape7[:,1:], "","Wavelength [$\mu$m]", "Transmittance",
       label=colSelect[1:],legendAlpha=0.5, pltaxis=[0.4,1, 0, 1]);


I am working on the file "08-Workshop-Overview/.ipynb_checkpoints/Workshop_Overview-checkpoint.ipynb".
The first task is:
Lesson 2: Basics of Python
Can you write Python code for it?


I am working on the file "08-Workshop-Overview/.ipynb_checkpoints/Workshop_Overview-checkpoint.ipynb".
The first task is:
Lesson 3: Operationalizing
Can you write Python code for it?

import pandas
dialogue_df = pandas.read_csv('../03-Operationalizing/antigone_dialogue.csv', index_col=0)
dialogue_tokens = [character.split() for character in dialogue_df['DIALOGUE']]
dialogue_len = [len(tokens) for tokens in dialogue_tokens]
dialogue_df['WORDS_SPOKEN'] = dialogue_len
dialogue_df = dialogue_df.sort_values('WORDS_SPOKEN', ascending = False)
# Let's visualize!

# Tells Jupyter to produce images in notebook
% pylab inline

# Makes images look good
style.use('ggplot')
dialogue_df['WORDS_SPOKEN'].plot(kind='bar')

###Who is the main protagonist? Maybe not Antigone?


I am working on the file "08-Workshop-Overview/.ipynb_checkpoints/Workshop_Overview-checkpoint.ipynb".
The first task is:
Lesson 4: Discriminating Words
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer

df = pandas.read_csv("../04-Discriminating-Words/BDHSI2016_music_reviews.csv", sep = '\t')

tfidfvec = TfidfVectorizer()
#create the dtm, but with cells weigthed by the tf-idf score.
dtm_tfidf_df = pandas.DataFrame(tfidfvec.fit_transform(df.body).toarray(), columns=tfidfvec.get_feature_names(), index = df.index)

df_genre = df['genre'].to_frame()
merged_df = df_genre.join(dtm_tfidf_df, how = 'right', lsuffix='_x')

#pull out the reviews for three genres, Rap, Alternative/Indie Rock, and Jazz
dtm_rap = merged_df[merged_df['genre_x']=="Rap"]
dtm_indie = merged_df[merged_df['genre_x']=="Alternative/Indie Rock"]
dtm_jazz = merged_df[merged_df['genre_x']=="Jazz"]

#print the words with the highest tf-idf scores for each genre
print("Rap Words")
print(dtm_rap.max(numeric_only=True).sort_values(ascending=False)[0:20])
print()
print("Indie Words")
print(dtm_indie.max(numeric_only=True).sort_values(ascending=False)[0:20])
print()
print("Jazz Words")
print(dtm_jazz.max(numeric_only=True).sort_values(ascending=False)[0:20])

###What words are distinct to reviews of Rap albums, Indie albums, and Jazz albums?
##Notice the word weights for the Rap albums compared to others. Are these reviews more different than other reviews?


I am working on the file "08-Workshop-Overview/.ipynb_checkpoints/Workshop_Overview-checkpoint.ipynb".
The first task is:
Lesson 5: Sentiment Analysis using the Dictionary Method
Can you write Python code for it?

pos_sent = open("../05-Dictionary-Method/positive_words.txt").read()
neg_sent = open("../05-Dictionary-Method/negative_words.txt").read()

positive_words=pos_sent.split('\n')
negative_words=neg_sent.split('\n')

text1_pos = [word for word in text1_tokens_clean if word in positive_words]
text2_pos = [word for word in text2_tokens_clean if word in positive_words]

text1_neg = [word for word in text1_tokens if word in negative_words]
text2_neg = [word for word in text2_tokens if word in negative_words]

print("Postive words in Melville")
print(len(text1_pos)/len(text1_tokens))
print()
print("Negative words in Melville")
print(len(text1_neg)/len(text1_tokens))
print()
print("Postive words in Austen")
print(len(text2_pos)/len(text2_tokens))
print()
print("Negative words in Austen")
print(len(text2_neg)/len(text2_tokens))

## Who is more postive, Melville or Austen?
## Melville has a similar precentage of postive and negative words (a whale is a whale, neither good nor bad)
## Austen is decidedly more positive than negative (it's the gentleman thing to do)


I am working on the file "08-Workshop-Overview/.ipynb_checkpoints/Workshop_Overview-checkpoint.ipynb".
The first task is:
Lesson 6: Topic Modeling
Can you write Python code for it?

import gensim
import pandas
from nltk.corpus import stopwords, words

metadata_df = pandas.read_csv('../07-Topic Modeling/txtlab_Novel150_English.csv')
fiction_path = '../07-Topic Modeling/txtalb_Novel150_English/'
novel_list = [open(fiction_path+file_name).read() for file_name in metadata_df['filename']]
novel_tokens_list = [novel.lower().split() for novel in novel_list]
dictionary = gensim.corpora.dictionary.Dictionary(novel_tokens_list)
proper_names = [word.lower() for word in words.words() if word.istitle()]
noise_tokens = [word for word in dictionary.values() if word.isalpha()==False or len(word)<=2]
bad_words = stopwords.words('english') + proper_names + noise_tokens
stop_ids = [_id for _id, count in dictionary.doc2bow(bad_words)]
dictionary.filter_tokens(bad_ids = stop_ids)
dictionary.filter_extremes(no_below = 40)
corpus = [dictionary.doc2bow(text) for text in novel_tokens_list]
lda_model = gensim.models.LdaModel(corpus, num_topics=25, alpha='auto', id2word=dictionary, iterations=2500, passes = 4)
list_of_doctopics = [lda_model.get_document_topics(text, minimum_probability=0) for text in corpus]
list_of_probabilities = [[probability for label,probability in distribution] for distribution in list_of_doctopics]
proba_distro_df = pandas.DataFrame(list_of_probabilities)
metadata_df = pandas.concat([metadata_df, pandas.DataFrame(list_of_probabilities)], axis=1)
annual_means_df = metadata_df.groupby('date').mean()

annual_means_df[8].plot(kind='bar', figsize=(8,8))

lda_model.show_topic(8)


I am working on the file "08-Workshop-Overview/Workshop Overview.ipynb".
The first task is:
Lesson 2: Basics of Python
Can you write Python code for it?


I am working on the file "08-Workshop-Overview/Workshop Overview.ipynb".
The first task is:
Lesson 4: Discriminating Words
Can you write Python code for it?

# Prep
from sklearn.feature_extraction.text import TfidfVectorizer

# Spreadsheet with Album Reviews
df = pandas.read_csv("../04-Discriminating-Words/BDHSI2016_music_reviews.csv", sep = '\t')

# Create a Document-Term Matrix (DTM), weight cells by Tf-Idf
tfidfvec = TfidfVectorizer()
dtm_tfidf_df = pandas.DataFrame(tfidfvec.fit_transform(df.body).toarray(), columns=tfidfvec.get_feature_names(), index = df.index)

# Add a Column for Album Genre
df_genre = df['genre'].to_frame()
merged_df = df_genre.join(dtm_tfidf_df, how = 'right', lsuffix='_x')

# Get Reviews for 3 Genres: Rap, Alternative/Indie Rock, and Jazz
dtm_rap = merged_df[merged_df['genre_x']=="Rap"]
dtm_indie = merged_df[merged_df['genre_x']=="Alternative/Indie Rock"]
dtm_jazz = merged_df[merged_df['genre_x']=="Jazz"]

# Print Words with Highest Tf-Idf Scores per Genre
print("Rap Words")
print(dtm_rap.max(numeric_only=True).sort_values(ascending=False)[0:20])
print()
print("Indie Words")
print(dtm_indie.max(numeric_only=True).sort_values(ascending=False)[0:20])
print()
print("Jazz Words")
print(dtm_jazz.max(numeric_only=True).sort_values(ascending=False)[0:20])

## What words are distinct to reviews of Rap albums, Indie albums, and Jazz albums?


I am working on the file "08-Workshop-Overview/Workshop Overview.ipynb".
The first task is:
Lesson 5: Sentiment Analysis using the Dictionary Method
Can you write Python code for it?

# Read Files with Positive & Negative Words
positive_file = open("../05-Dictionary-Method/positive_words.txt").read()
negative_file = open("../05-Dictionary-Method/negative_words.txt").read()

# Tokenize Lists of Words
positive_words = set(positive_file.split('\n'))
negative_words = set(negative_file.split('\n'))

# Get Positive Words from the Novels in Lesson 1
text1_pos = [word for word in text1_tokens_clean if word in positive_words]
text2_pos = [word for word in text2_tokens_clean if word in positive_words]

# Get their Negative Words
text1_neg = [word for word in text1_tokens if word in negative_words]
text2_neg = [word for word in text2_tokens if word in negative_words]

print("Melville")
print("% Positive Words:", np.round(len(text1_pos)/len(text1_tokens),4)*100)
print("% Negative Words:", np.round(len(text1_neg)/len(text1_tokens),4)*100)
print()
print("Austen")
print("% Positive Words:", np.round(len(text2_pos)/len(text2_tokens),4)*100)
print("% Negative Words:", np.round(len(text2_neg)/len(text2_tokens),4)*100)

## Who is more postive, Melville or Austen?


I am working on the file "08-Workshop-Overview/Workshop Overview.ipynb".
The first task is:
Lesson 7: Word Embedding
Can you write Python code for it?

import gensim
from nltk.tokenize import sent_tokenize

# Custom Tokenizer for Classroom Use
def fast_tokenize(text):
    return "".join([char for char in text.lower() if char not in string.punctuation]).split()

# Set Location of Corpus Folder
fiction_folder = '../07-Word2Vec/txtlab_Novel450_English/'

# Collect Text of Each Novel in 'fiction_folder'
novel_list = [open(fiction_folder + filename, 'r').read() for filename in os.listdir(fiction_folder)]

# Split Each Novel into Sentences
sentences = [sentence for novel in novel_list for sentence in sent_tokenize(novel)]

# Split Each Sentence into Tokens
words_by_sentence = [fast_tokenize(sentence) for sentence in sentences]

# Remove Any Sentences that Contain Zero Tokens
words_by_sentence = [sentence for sentence in words_by_sentence if sentence != []]

# Train word2vec model from txtLab corpus
model = gensim.models.Word2Vec(words_by_sentence, size=100, window=5, \
                               min_count=25, sg=1, alpha=0.025, iter=5, batch_words=10000)

# King - Man + Woman -> Queen
model.most_similar(positive=['woman', 'king'], negative=['man'])

## Man is to King as Woman is to ??
## What has our model learned about gender?


I am working on the file "08-dados_alunos.ipynb".
The first task is:
2) A entrada para objetos datetime é int
Can you write Python code for it?

new_date = []

for d in date.split():
    new_date.append(int(d))

new_date


I am working on the file "08-dados_alunos.ipynb".
The first task is:
4) Criando uma função que recebe o string de datas e retorna o objeto datetime
Can you write Python code for it?

def convert(date):
    return datetime(*map(int, date.split()))

convert(date)


I am working on the file "08-dados_alunos.ipynb".
The first task is:
Como converter o conjunto de números da coluna tempo para objetos datetime?
1) O que temos na coluna tempo?
Can you write Python code for it?

date = df['tempo'].ix[0]
date

date.split()


I am working on the file "08-dados_alunos.ipynb".
The first task is:
Generalized Extreme Value distribution
Can you write Python code for it?

import wafo.kdetools as wk

fig, ax = plt.subplots(figsize=(9, 9))

wk.TKDE(df['Hsig'], L2=0.5)(output='plot').plot('c--')
gev.plotepdf(symb1='g-')
wei.plotepdf(symb1='r-')

from scipy import stats
import matplotlib.pyplot as plt

def make_plot():
    fig, ax = plt.subplots(figsize=(9, 9))
    ax.hist(df['Hsig'], bins=10, normed=True, alpha=0.5);

    x = np.linspace(df['Hsig'].min(), df['Hsig'].max(), 1000)

    # Exponential Weibull
    params = stats.exponweib.fit(df['Hsig'])
    weib = stats.exponweib.pdf(x, *params)
    ax.plot(x, weib, label='exponweib')

    # Weibull minimum (Frechet right)
    params = stats.weibull_min.fit(df['Hsig'])
    weib = stats.weibull_min.pdf(x, *params)
    ax.plot(x, weib, label='weibull_min')

    # By hand!
    def weib(x, c):
        return c*x**(c-1)*np.exp(-x**c)
    c, loc, scale = params
    y = (x-loc) / scale
    w = weib(y, c) / scale
    ax.plot(x, w, '--', label='Manual')

    leg = ax.legend()

make_plot()


I am working on the file "08-dados_alunos.ipynb".
The first task is:
O último passo e usar essa nova coluna com o index
Can you write Python code for it?

df.set_index('tempo', inplace=True)

df.tail()


I am working on the file "08-dados_alunos.ipynb".
The first task is:
Pronto!  Agora podemos passar essa função para o pandas.read_excel e os valores serão convertidos automagicamente
Can you write Python code for it?

df = pd.read_excel("./data/2005.02_onda.xlsx",
                   parse_cols="A:C",
                   converters={'tempo': convert})

df.head()


I am working on the file "08-dados_alunos.ipynb".
The first task is:
Weibull
$$ c x^{c-1} e^{-x^c} $$
Can you write Python code for it?

c, loc, scale = stats.weibull_min.fit(df['Hsig'])

c, loc, scale

df['Hsig'].describe()[['max', '75%', 'mean', 'min', 'std']]


I am working on the file "08-dados_alunos.ipynb".
The first task is:
série temporal das alturas significativas e,
gráfico polar mostrando direção de onda e altura significativa
Can you write Python code for it?

%matplotlib inline

import matplotlib
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(11, 3.25))
df.plot(ax=ax, secondary_y=['Hsig']);

from oceans import stick_plot
from oceans.ff_tools.ocfis import pol2cart

x, y = pol2cart(df['dir'], df['Hsig'], units='deg')

fig, ax = plt.subplots(figsize=(11, 3.25))

q = stick_plot(df.index.to_pydatetime(), x.values, y.values, ax=ax)

ref = 1
qk = plt.quiverkey(q, 0.1, 0.85, ref,
                  "%s m" % ref,
                  labelpos='N', coordinates='axes')

import numpy as np

r = np.deg2rad(df['dir'])

fig, ax = plt.subplots(subplot_kw=dict(polar=True))

ax.plot(r, df['Hsig']);

from windrose import WindroseAxes
from matplotlib import pyplot as plt
import matplotlib.cm as cm

ax = WindroseAxes.from_ax()
ax.contourf(df['dir'], df['Hsig'], blowto=True, cmap=cm.RdBu_r)
ax.set_legend()

ax = WindroseAxes.from_ax()
ax.box(df['dir'], df['Hsig'], bins=np.arange(0, 1.8, 0.15))
ax.set_legend()

plt.hist(df['dir'], bins=15);

plt.hist(df['Hsig'], bins=15);


I am working on the file "08-dados_alunos.ipynb".
The first task is:
valores de máximo e minimo da altura significativa e,
média da altura de onda
Can you write Python code for it?


I am working on the file "08-inflacion.ipynb".
The first task is:
Ejemplo.-- Considere un flujo de caja constante de $ 1000. Conviértalo a dinero del período 0 usando una inflación inicial del 5% por período. La inflación cambia al 10% a partir del período 5.
Can you write Python code for it?

## flujo de caja
x = cf.cashflow(const_value=[1000]*10, start='2016')
cf.textplot(x)

## inflación
inflation = cf.interest_rate(const_value=[5]*10, start='2016', chgpts={5:10})
cf.textplot(inflation)

cf.const2curr(cflo=x, inflation=inflation)


I am working on the file "08-inflacion.ipynb".
The first task is:
Inflación
Se define como la pérdida de poder adquisitivo de la moneda. Existen diferentes medidas:

Indice de precios del consumidos (IPC).
Indice de precios del productor (IPP).
Medidas derivadas a partir del producto interno bruto.

Tasa promedio de inflación: tasa porcentual anual que representa el incremento de los precios sobre un período de un año respecto al año anterior.
Tratamiento de la inflación

Precios constantes.
Precios corrientes.
Tasa de interés del mercado $(i)$. Incluye una componente de utilidad económica y una de inflación.
Tasa de interés libre de inflación $(i')$. Representa únicamente la componente de utilidad económica.
Tasa de inflación general $(f)$. Incremento promedio anual en los precios de los bienes y servicios.

Relaciones de equivalencia

$F_n'$: valor en moneda constante (en precios del período 0) que ocurren en el flujo de caja al final del período $n$.

$F_n$: valor en moneda corriente que ocurren en el flujo de caja al final del período $n$.

Conversión entre moneda constante y corriente:
$$F_n=F_n^{'}~(1+f)^n$$
Valor presente:
$$P=F_n^{'}~(1+i^{'})^{-n} = F_n^{'}~(1+i)^{-n}$$
Relación entre tasas:
$$i^{'}=\frac{i-f}{1+f}$$
$$(1+i^{'})(1+f)=1+i$$
const2curr(cflo, inflation, base_date=0)
curr2const(cflo, inflation, base_date=0)
Convierte un flujo de efectivo en corrientes a constante y viceversa.

cflo -- flujo de efectivo.
inflation -- tasa de inflación por período.
base_date -- período base para realizar los cálculos.

Ejemplo.-- Considere un flujo de caja constante de $ 1000. Conviértalo a dinero del período 0 usando una inflación del 5% por período.
Can you write Python code for it?

x = cf.cashflow(const_value=[1000]*10, start='2016')
x

cf.const2curr(cflo=x,
              inflation=cf.interest_rate(const_value=[5]*10, start='2016'))


I am working on the file "08-model-monitoring.ipynb".
The first task is:
2. Create Monitoring Job
Retrieve the Vertex dataset and endpoint models to monitor
Can you write Python code for it?

dataset = vertex_ai.TabularDataset.list(
    filter=f"display_name={DATASET_DISPLAY_NAME}",
    order_by="update_time")[-1]

bq_source_uri = dataset.gca_resource.metadata["inputConfig"]["bigquerySource"]["uri"]

endpoint = vertex_ai.Endpoint.list(
    filter=f'display_name={ENDPOINT_DISPLAY_NAME}',
    order_by="update_time")[-1]

endpoint_uri = endpoint.gca_resource.name

model_ids = [model.id for model in endpoint.list_models()]


I am working on the file "08-model-monitoring.ipynb".
The first task is:
3. List Monitoring Jobs
Can you write Python code for it?

monitoring_jobs = job_client_beta.list_model_deployment_monitoring_jobs(parent=PARENT)
monitoring_job = [entry for entry in monitoring_jobs if entry.display_name == MONITORING_JOB_NAME][0]
monitoring_job


I am working on the file "08-model-monitoring.ipynb".
The first task is:
4. Simulate skewed prediction requests
Can you write Python code for it?

num_requests = 100

print("Simulation started...")
for idx in range(num_requests):
    request = [{
        "dropoff_grid": ["POINT(-87.6 41.9)"],
        "euclidean": [2064.2696],
        "loc_cross": [""],
        "payment_type": ["Credit Card"],
        "pickup_grid": ["POINT(-87.6 41.9)"],
        "trip_miles": [1.37],
        "trip_day": [int(random.uniform(10, 50))],
        "trip_hour": [int(random.uniform(10, 50))],
        "trip_month": [int(random.uniform(1, 10))],
        "trip_day_of_week": [int(random.uniform(1, 7))],
        "trip_seconds": [int(random.uniform(60, 600))]
    }]

    endpoint.predict(request)
    time.sleep(0.5)

    if idx % 10 == 0:
        print(f'{idx + 1} of {num_requests} prediction requests were invoked.')
print("Simulation finished.")


I am working on the file "08-model-monitoring.ipynb".
The first task is:
Configure the monitoring job
Can you write Python code for it?

skew_thresholds = {
    feature: vertex_ai_beta.ThresholdConfig(value=float(value))
    for feature, value in SKEW_THRESHOLDS.items()
}

drift_thresholds = {
    feature: vertex_ai_beta.ThresholdConfig(value=float(value))
    for feature, value in DRIFT_THRESHOLDS.items()
}

skew_config = vertex_ai_beta.ModelMonitoringObjectiveConfig.TrainingPredictionSkewDetectionConfig(
    skew_thresholds=skew_thresholds
)

drift_config = vertex_ai_beta.ModelMonitoringObjectiveConfig.PredictionDriftDetectionConfig(
    drift_thresholds=drift_thresholds
)

sampling_config = vertex_ai_beta.SamplingStrategy(
    random_sample_config=vertex_ai_beta.SamplingStrategy.RandomSampleConfig(
        sample_rate=LOG_SAMPLE_RATE
    )
)

schedule_config = vertex_ai_beta.ModelDeploymentMonitoringScheduleConfig(
    monitor_interval=Duration(seconds=MONITOR_INTERVAL)
)

training_dataset = vertex_ai_beta.ModelMonitoringObjectiveConfig.TrainingDataset(
    target_field=TARGET_FEATURE_NAME,
    bigquery_source = vertex_ai_beta.types.io.BigQuerySource(
        input_uri=bq_source_uri
    )
)

objective_template = vertex_ai_beta.ModelDeploymentMonitoringObjectiveConfig(
    objective_config=vertex_ai_beta.ModelMonitoringObjectiveConfig(
        training_dataset=training_dataset,
        training_prediction_skew_detection_config=skew_config,
        prediction_drift_detection_config=drift_config,
    )
)

deployment_objective_configs = []
for model_id in model_ids:
    objective_config = copy.deepcopy(objective_template)
    objective_config.deployed_model_id = model_id
    deployment_objective_configs.append(objective_config)

alerting_config = vertex_ai_beta.ModelMonitoringAlertConfig(
    email_alert_config=vertex_ai_beta.ModelMonitoringAlertConfig.EmailAlertConfig(
        user_emails=NOTIFY_EMAILS
    )
)


I am working on the file "08-model-monitoring.ipynb".
The first task is:
Delete Monitoring Job
Can you write Python code for it?


I am working on the file "08-model-monitoring.ipynb".
The first task is:
Set configurations
Can you write Python code for it?

DATASET_DISPLAY_NAME = 'chicago-taxi-tips'
ENDPOINT_DISPLAY_NAME = 'chicago-taxi-tips-classifier'
MONITORING_JOB_NAME = f"monitor-{ENDPOINT_DISPLAY_NAME}"
NOTIFY_EMAILS = ["<your-email-address>"] # Change to your email address.

LOG_SAMPLE_RATE = 0.8
MONITOR_INTERVAL = 3600
TARGET_FEATURE_NAME = 'tip_bin'


I am working on the file "08-model-monitoring.ipynb".
The first task is:
Submit the job for creation
Can you write Python code for it?

response = job_client_beta.create_model_deployment_monitoring_job(
    parent=PARENT, model_deployment_monitoring_job=job
)
response


I am working on the file "08-notebook.ipynb".
The first task is:
Hier sollte ausschließlich Keine Analyse möglich stehen:
Can you write Python code for it?


I am working on the file "08-notebook.ipynb".
The first task is:
Hier wurde versucht, Aufforderungssätze zu modellieren. Allerdings akzeptiert diese Grammatik immer noch viele ungrammatische Sätze.

Fügen Sie ein Merkmal MOOD mit den möglichen Werten imp und ind für Imperativ und Indikativ hinzu und deklarieren Sie die entsprechenden Werte im Lexikon (den lexikalischen Regeln).
Formulieren Sie dann Bedingungen in den syntaktischen Regeln, sodass folgende grammatische Sätze (pos_sentences) weiterhin akzeptiert und folgende ungrammatische Sätze (neg_sentences) abgelehnt werden.
Can you write Python code for it?

pos_sentences = [
    "das Kind schläft",
    "das Kind liest das Buch",
    "lies das Buch",
    "schlaf"
]

neg_sentences = [
    "das Kind schlaf",
    "das Kind lies das Buch",
    "liest das Buch",
    "schläft"
]

import nltk
from IPython.display import display
import sys

def test_grammar(grammar, sentences):
    cfg = nltk.grammar.FeatureGrammar.fromstring(grammar)
    parser = nltk.parse.FeatureEarleyChartParser(cfg)

    for i, sent in enumerate(sentences, 1):
        print("Satz {}: {}".format(i, sent))
        sys.stdout.flush()
        results = parser.parse(sent.split())
        analyzed = False
        for tree in results:
            display(tree)  # tree.draw() oder print(tree)
            analyzed = True
        if not analyzed:
            print("Keine Analyse möglich", file=sys.stderr)
            sys.stderr.flush()


I am working on the file "08-statistics.ipynb".
The first task is:
Basic statistical functions
numpy contains a number of basic statistical functions, such as min, max and mean. These will act on entire arrays to give the "all time" minimum, maximum, and average rainfall:
Can you write Python code for it?

print("Minimum rainfall: {}".format(rainfall.min()))
print("Maximum rainfall: {}".format(rainfall.max()))
print("Mean rainfall: {}".format(rainfall.mean()))


I am working on the file "08-statistics.ipynb".
The first task is:
Categorical data
Looking at the means by month, it would be better to give them names rather than numbers. We will also summarize the available information using a boxplot:
Can you write Python code for it?

months = ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
pyplot.boxplot(rainfall, labels=months)
pyplot.xlabel('Month')
pyplot.ylabel('Mean rainfall');


I am working on the file "08-statistics.ipynb".
The first task is:
If we wanted to plot the mean rainfall per year, across all years, this would be tedious - there are 145 years of data in the file. Even computing the mean rainfall in each month, across all years, would be bad with 12 months. We could write a loop. However, numpy allows us to apply a function along an axis of the array, which does this is one operation:
Can you write Python code for it?

mean_rainfall_in_month = rainfall.mean(axis=0)
mean_rainfall_per_year = rainfall.mean(axis=1)


I am working on the file "08-statistics.ipynb".
The first task is:
It looks like there's a good chance that the slight decrease in mean rainfall with time is a real effect.
Random numbers
Random processes and random variables may be at the heart of probability and statistics, but computers cannot generate anything "truly" random. Instead they can generate pseudo-random numbers using random number generators (RNGs). Constructing a random number generator is a hard problem and wherever possible you should use a well-tested RNG rather than attempting to write your own.
Python has many ways of generating random numbers. Perhaps the most useful are given by the numpy.random module, which can generate a numpy array filled with random numbers from various distributions. For example:
Can you write Python code for it?

from numpy import random

uniform = random.rand(10000)
normal = random.randn(10000)

fig = pyplot.figure()
ax1 = fig.add_subplot(1,2,1)
ax2 = fig.add_subplot(1,2,2)
ax1.hist(uniform, 20)
ax1.set_title('Uniform data')
ax2.hist(normal, 20)
ax2.set_title('Normal data')
fig.tight_layout()
fig.show();


I am working on the file "08-statistics.ipynb".
The first task is:
It looks like there's no significant change in mean rainfall with time.
Random numbers
Random processes and random variables may be at the heart of probability and statistics, but computers cannot generate anything "truly" random. Instead they can generate pseudo-random numbers using random number generators (RNGs). Constructing a random number generator is a hard problem and wherever possible you should use a well-tested RNG rather than attempting to write your own.
Python has many ways of generating random numbers. Perhaps the most useful are given by the numpy.random module, which can generate a numpy array filled with random numbers from various distributions. For example:
Can you write Python code for it?


I am working on the file "08-statistics.ipynb".
The first task is:
More distributions
Whilst the standard distributions are given by the convenience functions above, the full documentation of numpy.random shows many other distributions available. For example, we can draw $10,000$ samples from the Beta distribution using the parameters $\alpha = 1/2 = \beta$ as
Can you write Python code for it?

beta_samples = random.beta(0.5, 0.5, 10000)

pyplot.hist(beta_samples, 20)
pyplot.title('Beta data')
pyplot.show();


I am working on the file "08-statistics.ipynb".
The first task is:
Much better ways of working with categorical data are available through more specialized packages.
Regression
We can go beyond the basic statistical functions in numpy and look at other standard tasks. For example, we can look for simple trends in our data with a linear regression. There is a function to compute the linear regression in scipy we can use. We will use this to see if there is a trend in the mean yearly rainfall:
Can you write Python code for it?

from scipy import stats

slope, intercept, r_value, p_value, std_err = stats.linregress(years, mean_rainfall_per_year)

pyplot.plot(years, mean_rainfall_per_year, 'b-', label='Data')
pyplot.plot(years, intercept + slope*years, 'k-', label='Linear Regression')
pyplot.xlabel('Year')
pyplot.ylabel('Mean rainfall')
pyplot.legend();

print("The change in rainfall (the slope) is {}.".format(slope))
print("However, the error estimate is {}.".format(std_err))
print("The probability that the rainfall is correlated"
      " with the year is {}.".format(r_value))
print("The probability that the slope is zero is {}.".format(p_value))


from scipy import stats

slope, intercept, r_value, p_value, std_err = stats.linregress(years, mean_rainfall_per_year)

pyplot.plot(years, mean_rainfall_per_year, 'b-', label='Data')
pyplot.plot(years, intercept + slope*years, 'k-', label='Linear Regression')
pyplot.xlabel('Year')
pyplot.ylabel('Mean rainfall')
pyplot.legend();

print("The change in rainfall (the slope) is {}.".format(slope))
print("However, the error estimate is {}.".format(std_err))
print("The correlation coefficient between rainfall and year"
      " is {}.".format(r_value))
print("The probability that the slope is zero is {}.".format(p_value))


I am working on the file "08-statistics.ipynb".
The first task is:
Of more interest would be either

the mean (min/max) rainfall in a given month for all years, or
the mean (min/max) rainfall in a given year for all months.

So the mean rainfall in the first year, 1855, would be
Can you write Python code for it?

print ("Mean rainfall in 1855: {}".format(rainfall[0, :].mean()))


I am working on the file "08-statistics.ipynb".
The first task is:
The axis argument gives the direction we want to keep - that we do not apply the operation to. For this data set, each row contains a year and each column a month. To find the mean in a given month we want to keep the row information (axis 0) and take the mean over the column. To find the mean in a given year we want to keep the column information (axis 1) and take the mean over the row.
We can now plot how the mean varies with each year.
Can you write Python code for it?

pyplot.plot(years, mean_rainfall_per_year)
pyplot.xlabel('Year')
pyplot.ylabel('Mean rainfall');


I am working on the file "08-statistics.ipynb".
The first task is:
This isn't particularly pretty or clear: a nicer example would use better packages, but a quick fix uses an alternative matplotlib approach:
Can you write Python code for it?

pyplot.plot(years, mean_rainfall_per_year)
pyplot.fill_between(years, mean_rainfall_per_year - std_rainfall_per_year,
                    mean_rainfall_per_year + std_rainfall_per_year,
                    alpha=0.25, color=None)
pyplot.xlabel('Year')
pyplot.ylabel('Mean rainfall');


I am working on the file "08-statistics.ipynb".
The first task is:
We can also compute the standard deviation:
Can you write Python code for it?


I am working on the file "08-statistics.ipynb".
The first task is:
We can do this $5,000$ times and compute the mean of each set of samples:
Can you write Python code for it?

n_trials = 5000
beta_means = numpy.zeros((n_trials,))

for trial in range(n_trials):
    beta_samples = random.beta(0.5, 0.5, 10000)
    beta_means[trial] = numpy.mean(beta_samples)

pyplot.hist(beta_means, 20)
pyplot.title('Mean of Beta trials')
pyplot.show();


I am working on the file "08-statistics.ipynb".
The first task is:
We can now plot, for example, the rainfall in January for all years:
Can you write Python code for it?

%matplotlib inline
from matplotlib import rcParams
rcParams['figure.figsize']=(12,9)

from matplotlib import pyplot

pyplot.plot(years, rainfall[:,0])
pyplot.xlabel('Year')
pyplot.ylabel('Rainfall in January');


I am working on the file "08-statistics.ipynb".
The first task is:
We can then add confidence intervals to the plot:
Can you write Python code for it?

pyplot.errorbar(years, mean_rainfall_per_year, yerr = std_rainfall_per_year)
pyplot.xlabel('Year')
pyplot.ylabel('Mean rainfall');


I am working on the file "08-statistics.ipynb".
The first task is:
We can use numpy to load this data into a variable, where we can manipulate it. This is not ideal: it will lose the information in the header, and that the first column corresponds to years. However, it is simple to use.
Can you write Python code for it?

import numpy

data = numpy.loadtxt('southampton_precip.txt')

data


I am working on the file "08-statistics.ipynb".
The first task is:
We see that the first column - the year - has been converted to a floating point number, which is not helpful. However, we can now split the data using standard numpy operations:
Can you write Python code for it?

years = data[:, 0]
rainfall = data[:, 1:]


I am working on the file "08-statistics.ipynb".
The first task is:
Whilst the mean rainfall in January, averaging over all years, would be
Can you write Python code for it?

print ("Mean rainfall in January: {}".format(rainfall[:, 0].mean()))


I am working on the file "08-text-mining.ipynb".
The first task is:
Feature Extraction
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer

text = pd.read_csv('./Datasets/text_mining.csv').drop(['document_id','direction'], axis =1)
text2 =text[(text['category_id'] == 1) | (text['category_id'] == 8)  ]

text[text['category_id'] == 1].head()

text[text['category_id'] == 8].head()

climate_con = text[text['category_id'] == 1].iloc[0:5]
politics = text[text['category_id'] == 8].iloc[0:5]

countvectorizer = CountVectorizer()

cli = ''
for i in climate_con.text.as_matrix(): cli = cli + ' ' + i
pol = ''
for i in politics.text.as_matrix(): pol = pol + ' ' + i

content = [cli, pol]

countvectorizer.fit(content)

doc_vec = countvectorizer.transform(content)

df = pd.DataFrame(doc_vec.toarray().transpose(), index = countvectorizer.get_feature_names())

df.sort_values(0, ascending=False)

df.sort_values(1, ascending=False)

tfidf = TfidfVectorizer()

tfidf_vec = tfidf.fit_transform(content)

df2 = pd.DataFrame(tfidf_vec.toarray().transpose(), index =tfidf.get_feature_names())

df2.sort_values(0, ascending=False)

df2.sort_values(1, ascending=False)

tfidf.vocabulary_


I am working on the file "08-text-mining.ipynb".
The first task is:
Naive Bayes

Naive Bayes methods are a set of supervised learning algorithms based on applying Bayes’ theorem with the “naive” assumption of independence between every pair of features.

Bayes Rule:
$$ P(Y|X) = \frac{P(Y)P(X|Y)}{P(X)} $$
and Naive assumption:
$$ P(x_i | y,x_1,x_2,...x_{i-1},x_{i+1},...,x_n) = P(x_i|y) $$
leads to:
$$ P(y| x_1,x_2,...,x_n) = \frac{P(y) \prod_{i=1}^n P(x_i|y)}{P(x_1,x_2,...,x_n)} $$
If the purpose is only to classify:
$$ \hat{y} = arg\max_y P(y)\prod_{i=1}^n P(x_i|y) $$
Can you write Python code for it?

from sklearn.naive_bayes import GaussianNB

model = GaussianNB()

model.fit(tfidf.transform(text2.text).toarray(), text2.category_id)

model.sigma_.shape

from sklearn.metrics import classification_report
print(classification_report(text2.category_id, model.predict(tfidf.transform(text2.text).toarray())))


I am working on the file "08-text-mining.ipynb".
The first task is:
Stemming
Can you write Python code for it?

from nltk.stem import SnowballStemmer

stemmer = SnowballStemmer('english')

stemmer.stem("impressive")

stemmer.stem("impressness")

from hazm import Stemmer

stem2 = Stemmer()

stem2.stem('کتاب ها')

stem2.stem('کتاب‌هایش')

stem2.stem('کتاب هایم')


I am working on the file "08-text-mining.ipynb".
The first task is:
stopword
Can you write Python code for it?

from nltk.corpus import stopwords

stopwords.words('english')[:10]

len(stopwords.words())

import hazm
hazm.stopwords_list()

len(hazm.stopwords_list())


I am working on the file "08. 미적분과 최적화/01. SymPy를 사용한 함수 미분.ipynb".
The first task is:
복수의 기호를 사용하는 경우에도 편미분을 해야 한다.
Can you write Python code for it?

x, mu, sigma = sympy.symbols('x mu sigma')
f = sympy.exp((x-mu)**2)/sigma**2
f

sympy.diff(f, x)

sympy.simplify(sympy.diff(f, x))


I am working on the file "08. 미적분과 최적화/01. SymPy를 사용한 함수 미분.ipynb".
The first task is:
역함수(inverse function)는 함수의 입력과 출력을 반대로 한 것이며 다음과 같은 기호로 표시한다.
$$ y = f(x), \;\;\; \rightarrow \;\;\; x = f^{-1}(y) $$
예측 문제와 함수
예측(prediction) 문제는 독립 변수, 혹은 feature $x$를 입력으로 하여 원하는 종속 변수 혹은 targer $y$와 가능한한 비슷한 값을 만드는 함수 $f$를 찾는 문제라고 할 수 있다.
$$ y \approx \hat{y} = f(x) $$
데이터 분석에서 많이 사용되는 함수들
데이터 분석에서 많이 사용되는 함수의 형태는 다항식(polynomial) 함수, 지수(exponential) 함수, 로그(log) 함수 등이다.
다항식 함수
다항식 함수는 상수항 $c_0$, 일차항 $c_1x$, 이차항 $c_2x^2$, $\cdots$ 등의 거듭제곱 항의 선형 조합으로 이루어진 함수이다. 다음은 단변수(uni-variate) 다항식 함수의 전형적인 형태이다.
$$ f(x) = c_0 + c_1 x + c_2 x^2 + \cdots + c_n x^n $$
지수 함수와 로그 함수
밑(base)를 오일러 수 $e$로 하는 지수함수는 다음과 같이 표시한다. 이는 $e$라는 숫자를 $x$번 거듭제곱한 것이라 생각하면 된다.
$$ y = e^x $$
또는
$$ y = \exp x $$
지수 함수의 역함수는 자연로그 함수이다.
$$ y = \log x $$
만약 밑이 $e$가 아닌 경우에는 다음과 같이 변형하여 사용한다.
$$ y = a^x = e^{\log a \cdot x} $$
함수의 그래프와 기울기
함수의 형상을 직관적으로 파악하기 위해 그래프(graph)를 사용하기도 한다. 파이썬에서는 matplotlib의 라인 플롯을 사용하여 그래프를 만들 수 있다.
다만 matplotlib에서는 구체적인 위치가 있어야지만 플롯을 만들 수 있기 때문에 그래프를 작성할 $x$ 영역을 작은 구간으로 나눈 벡터를 생성하고 이 벡터 값에 대한 함수값을 계산하여 그래프를 작성한다. 구간의 간격이 너무 크면 그래프가 부정확해지고 구간의 간격이 너무 작으면 쓸데없이 세부적인 그림을 그리게 되므로 계산 시간이 증가하고 메모리 등의 리소스가 낭비된다.
Can you write Python code for it?


I am working on the file "08. 미적분과 최적화/01. SymPy를 사용한 함수 미분.ipynb".
The first task is:
일단 심볼 변수를 정의하면 이를 사용하여 다음과 같이 함수를 정의한다. 이 때 수학 함수는 SymPy 전용 함수를 사용해야 한다.
Can you write Python code for it?


I am working on the file "08. 미적분과 최적화/01. SymPy를 사용한 함수 미분.ipynb".
The first task is:
함수가 정의되면 diff 명령으로 미분을 할 수 있다. 또한 simplify 명령으로 소인수분해 등을 통한 수식 정리가 가능하다.
Can you write Python code for it?


I am working on the file "08. 미적분과 최적화/01. SymPy를 사용한 함수 미분.ipynb".
The first task is:
함수의 그래프는 앞에서 그린 것처럼 부드러운 곡선(curve)의 형태로 나타나는 경우가 많다. 이 곡선에 대해 한 점만 공통으로 가지는 접선(tangent)를 그릴 수 있는데 이 접선이 수평선과 이루는 각도를 기울기(slope)라고 한다.
Can you write Python code for it?


I am working on the file "08.Let_it_Function.ipynb".
The first task is:
As with the print statement, you can perform this in a single line:
Can you write Python code for it?

def sq_num2(num):
    return num**2

sq_num2(5)


I am working on the file "08.Let_it_Function.ipynb".
The first task is:
Best Practices in Importing
First off, what I refer to as libraries, others might also refer to as Modules.
Next - it's generally bad to use "from library import *"
Quoting from the Google Python Style Guide:
``
Useimport xfor importing packages and modules.
Usefrom x import ywhere x is the package prefix and y is the module name with no prefix.
Usefrom x import y as z` if two modules named y are to be imported or if y is an inconveniently long name.
For example the module sound.effects.echo may be imported as follows:
from sound.effects import echo
...
echo.EchoFilter(input, output, delay=0.7, atten=4)
Do not use relative names in imports. Even if the module is in the same package, use the full package name. This helps prevent unintentionally importing a package twice.
```
Don't worry if you haven't understood all of that, I just want you to recall some of the keywords mentioned here when we start implementing these principles.
User Defined Functions
By now, you should have noticed a pattern. Every new technique we learn, is progressively more and more powerful. And we also combine many of the techniques we have already learnt.

User defined functions are simply a way for a user - you - to define code blocks that do exactly what you want. Don't want to write code to say "Hello User!" every time? Define a function!

As usual - very simple syntax.

def function_name(optional_input):
        inputs or variables
        functions like print
        a return function if needed

Let's break this down in our examples.
Can you write Python code for it?

# Function to say hello to the user
def say_hello():
    name = input("What's your name? ")
    print("Hello ", name,"!")

say_hello()

list_a = ["a", 1, 42, 19, "c", "23",1,2,3,4,5,6]
type(list_a)

for i in list_a:
    print(type(i))

len(list_a)

def list_scorer(list_name):
    if type(list_name) == list:
        print("Correctly identified a list of length,",len(list_name),"items.")
        for i in list_name:
            print(type(i))
    else:
        print("This is not a list.")

list_scorer(list_a)

list_scorer("Hello")


I am working on the file "08.Let_it_Function.ipynb".
The first task is:
Exercise
Here's a challenge for you. Can you modify the above code to make sure that a password is atleast 6 character, but not more than 20 character. The first letter must be a capital letter, and there has to be atleast 1 number. The code below can help with generating random capitalised characters, and numbers.
random.choice('ABCDEFGHIJKLMNOPQRSTUVXYZ')
random.choice('123456789')
It's ok to struggle with this - the more you see different patterns of code, the more you learn. Don't panic if you can't figure it out in 2 minutes. Start out with writing some pseudo-code in plain English (or your native language), tracing out how you would go about this.
Can you write Python code for it?


I am working on the file "08.Let_it_Function.ipynb".
The first task is:
For a data scientist, writing efficient functions can come in really handy during the data cleaning phase. Let's see one example.
Can you write Python code for it?

def clean_up(tel_num):
    result = ""
    digits = {"0","1","2","3","4","5","6","7","8","9"}
    for character in tel_num:
        if character in digits:
            result = result + character
    return result

client_phone_num = "+1-555-123-1234 Barone Sanitation (Call only day time)"
clean_up(client_phone_num)


I am working on the file "08.Let_it_Function.ipynb".
The first task is:
Let's also look at the 'return' statement. When you print something using the standard print syntax:

print("Hello World!")
print(2**2)
it only displays that output to the screen. So in the case of 2**2 which is 4, the value is merely displayed, but it cannot be used by the program for any purpose.
For that, we 'return' the value to the program.

Can you write Python code for it?

def sq_num(num):
    squared = num ** 2
    return squared

sq_num(10)


I am working on the file "08.Let_it_Function.ipynb".
The first task is:
One important thing while dealing with functions. What happens inside a function, stays in a function.
<img src="images/vegas.gif">
None of the variables defined inside a function can be called outside of it. Want to test it out?
Can you write Python code for it?

def cubist(num):
    cubed = num**3
    return cubed

cubist(3)

print(cubed) # Will give an error


I am working on the file "08.Let_it_Function.ipynb".
The first task is:
So as you can see, you have used a lot of functions already.
Now compare the Python code above to one way to handle a similar problem in C. Just to be clear, C is a very powerful langauage, so this isn't meant to show C in poor light, but to show you an area where Python is much more user friendly.
Importing from Libraries

This image above is from the awesome XKCD series. If you're not familiar with it, I encourage you to check them out, hilarious geek humour!

We will often have to import a lot of functions from existing libraries in Python.

Why? Because someone else very generously wrote hundreds or thousands of lines of codes so we don't have to. The large library base of Python is one of the reasons Data Scientists use it.
This is how the import syntax works:

import libraryName
import libraryName as ln
from libraryName import specificFunction
from libraryName import *

Let's see a few examples below.
Can you write Python code for it?

# Import the library
import random

# Initiate a for loo
for i in range(5):
# x is equal to a random value, that we got from the library named random, and the function named random()
    x = random.random()
    print(round(x,2))

# Circumference of a circle
from math import pi
radius = int(input("Enter the radius in cm: "))
c = 2*pi*radius
area = pi*(radius**2)
print("The circumference of the circle is: ",c)
print("The area of the circle is: ", area)


I am working on the file "08._Clustering_Hierarquico_Microarray.ipynb".
The first task is:
$Z$ é uma matriz $(n−1) \times4$
Clusters Z[i, 0] e Z[i, 1] são combinados para formar o cluster $i$.
A distância entre os clusters mesclados em $i$ é Z[i, 2]
O número de observações no cluster $i$ e Z[i,4]
Can you write Python code for it?

print Z[0:5]
print '...'
print Z[-6:-1]

Z.shape


I am working on the file "08._Clustering_Hierarquico_Microarray.ipynb".
The first task is:
A primeira linha contém o nome das colunas:
Can you write Python code for it?

cycles = reader.next()[1:]

print len(cycles)
print cycles[0:5]


I am working on the file "08._Clustering_Hierarquico_Microarray.ipynb".
The first task is:
Carregando os dados
O arquivo cellcycle.txt é uma tabela tipo TSV (tab separated values) com a informação de expressão gênica de 6178 genes:
Can you write Python code for it?

import csv
csvfile = open('data/cellcycle.txt', 'r')
reader = csv.reader(csvfile, delimiter='\t')


I am working on the file "08._Clustering_Hierarquico_Microarray.ipynb".
The first task is:
Exibição de um dendograma
Função que, para um índice $i$, devolve o nome do gene:
Can you write Python code for it?

llf = lambda id : genes[id]

plt.figure(figsize=(12, 30))
plt.title('Dendrograma')
plt.ylabel('Gene')
plt.xlabel(u'Distância (Ward)')
den = dendrogram(Z, orientation='left', leaf_font_size=9, leaf_label_func=llf)


I am working on the file "08._Clustering_Hierarquico_Microarray.ipynb".
The first task is:
Heatmap de genes aglomerados
Can you write Python code for it?

import seaborn as sns

plt.figure(2, figsize=(12,36))
hm = sns.heatmap(X[den['leaves'],:], square=True, cbar=False, xticklabels=False)
gene_labels = [genes[i] for i in den['leaves']]
ylabels = hm.set_yticklabels(gene_labels, rotation=0)


I am working on the file "08._Clustering_Hierarquico_Microarray.ipynb".
The first task is:
Já a primeira coluna contém o nome dos genes:
Can you write Python code for it?

genes = [row[0] for row in reader]

print len(genes)
print genes[0:5]

csvfile.close()


I am working on the file "08._Clustering_Hierarquico_Microarray.ipynb".
The first task is:
Para facilitar a visualização, vamos trabalhar com os primeiros 250 genes:
Can you write Python code for it?

data = np.genfromtxt('data/cellcycle.txt', delimiter='\t', skip_header=1,
                     usecols=range(1, 83), missing_values='', filling_values=0.)
X = data[0:250,:]
n, m = X.shape
print n, m

X

%matplotlib notebook
import matplotlib.pyplot as plt


I am working on the file "08/.ipynb_checkpoints/08-checkpoint.ipynb".
The first task is:
How old are the programmers that answered this survey?
Can you write Python code for it?


I am working on the file "08/08.ipynb".
The first task is:
How many full-stack web developers are male versus female?
Can you write Python code for it?

gender_df.groupby('Gender')['Occupation'].value_counts()
gender_df = gender_df[gender_df['Occupation'] == "Full-stack web developer"]
gender_df.groupby('Gender')['Occupation'].value_counts().plot(kind="bar", color = ['#599ad3', '#f9a65a'])

 #gender_df.groupby('Gender')['Years IT / Programming Experience'].value_counts().plot(kind="bar", color = ['#624ea7', '#599ad3', '#f9a65a', '#9e66ab', 'purple'])

df['Age'].value_counts()


I am working on the file "08/08.ipynb".
The first task is:
How many males answer this survey? How many females answered this survey?
Can you write Python code for it?

gender_df = df[(df['Gender'] == 'Male') | (df['Gender'] == 'Female')]
print(gender_df['Gender'].value_counts())


I am working on the file "08/08.ipynb".
The first task is:
How old are the programmers that answered this survey?
Can you write Python code for it?


I am working on the file "08/08.ipynb".
The first task is:
Is there a difference in year's of experience based on one's gender?
Can you write Python code for it?

gender_df.groupby('Gender')['Years IT / Programming Experience'].value_counts().sort_values().plot(kind="bar", color = ['#599ad3', '#f9a65a'])


I am working on the file "08/08.ipynb".
The first task is:
Job Distribution by Gender -- woah too much data
Can you write Python code for it?

df.groupby('Gender')['Occupation'].value_counts().plot(kind="bar", color = ['#599ad3', '#f9a65a']) # too mmuch data to appropriately display


I am working on the file "08/08.ipynb".
The first task is:
What is the age distribution by gender?
Can you write Python code for it?

gender_df.groupby('Gender')['Age'].value_counts().sort_values().plot(kind="bar", color = ['#624ea7', '#599ad3', '#f9a65a', '#9e66ab', 'purple'])

df["AgeScale"] = df["Age"].apply(str).replace("< 20", "0").apply(str).replace("20-24", "1").apply(str).replace("25-29", "2").apply(str).replace("30-34", "3").apply(str).replace("30-34", "3").apply(str).replace("35-39", "4").apply(str).replace("40-50", "5").apply(str).replace("51-60", "6").apply(str).replace("> 60", "7")

print(df["AgeScale"].head(10))

years_df =df[df['AgeScale'] != "Prefer not to disclose"]
years_df['AgeScale'] = years_df['AgeScale'].astype(float)

print(years_df).head()

years_df['Years IT / Programming Experience'].value_counts()

years_df['ExperienceRank'] = years_df['Years IT / Programming Experience'].apply(str).replace("Less than 1 year", "0").apply(str).replace("1 - 2 years", "1").apply(str).replace("2 - 5 years", "2").apply(str).replace("6 - 10 years", "3").apply(str).replace("11+ years", "4").astype(float)

# years_df.head()

years_df['ExperienceRank'].value_counts()

years_df['AgeScale'].value_counts()

#years_df['ExperienceRank'] = float(years_df['ExperienceRank'])
# years_df['AgeScale'] = float(years_df['AgeScale'])
# years_df['AgeScale'] = years_df['AgeScale'].apply(int)

#years_df['ExperienceRank'] = parseInt(years_df['ExperienceRank'])

#years_df['ExperienceRank'] = pd.Series(years_df['ExperienceRank'])
#years_df['AgeScale'] = pd.Series(years_df['AgeScale'])

moneyScatter = years_df.plot(kind='scatter', x='ExperienceRank', y='AgeScale', alpha=0.2) # caegorical data dos not display well on scatter plots
#moneyScatter.set_title('Distribution of Money Spent Amongst Respondents to the Survey by Age')
#moneyScatter.set_xlabel('Months Programming')
#moneyScatter.set_ylabel('Hours Spent Learning Each Week')
#plt.savefig('studyingovertime.png')

years_df['ExperienceRank'].describe()

years_df[['ExperienceRank','AgeScale']] = years_df[['ExperienceRank','AgeScale']].apply(pd.to_numeric)
# years_df.apply(lambda x: pd.to_numeric(x,...


I am working on the file "08/08.ipynb".
The first task is:
What is the least popular occupation?
Can you write Python code for it?


I am working on the file "08/08.ipynb".
The first task is:
What text editor do these individuals prefer?
Can you write Python code for it?

df['Preferred text editor'].value_counts().head(10).plot(kind="barh", color = ['#624ea7', '#599ad3', '#f9a65a', '#9e66ab', 'purple'])

df['Preferred text editor'].value_counts().head(10).plot(kind="bar", color = ['#624ea7', '#599ad3', '#f9a65a', '#9e66ab', 'purple'])

# df['Training & Education: BS in CS'].value_counts().head(10).plot(kind="bar", color = ['#624ea7', '#599ad3', '#f9a65a', '#9e66ab', 'purple'])


I am working on the file "08/H2B_Applications/.ipynb_checkpoints/H-2B_Visa_Programme_2015-checkpoint.ipynb".
The first task is:
1. How many requests did the Office of Foreign Labor Certification (OFLC) receive in 2015?
Can you write Python code for it?


I am working on the file "08/H2B_Applications/.ipynb_checkpoints/H-2B_Visa_Programme_2015-checkpoint.ipynb".
The first task is:
11. Which industries had the largest need?
Can you write Python code for it?

#changing df['NAIC_CODE'] from non_null object into int

#This .fillna(0.0) is magic. I found it here:
#http://stackoverflow.com/questions/21291259/convert-floats-to-ints-in-pandas
#df['NAIC_CODE'] = df['NAIC_CODE'].fillna(0.0).astype(int)
#But it turns out, it only works for my one fill. Not on the other. Why?


I am working on the file "08/H2B_Applications/.ipynb_checkpoints/H-2B_Visa_Programme_2015-checkpoint.ipynb".
The first task is:
2. How many jobs did that regard in total? And how many full time positions?
Can you write Python code for it?


I am working on the file "08/H2B_Applications/.ipynb_checkpoints/H-2B_Visa_Programme_2015-checkpoint.ipynb".
The first task is:
6. What was the most common unit of pay (daily, weekly, monthly)?
Can you write Python code for it?


I am working on the file "08/H2B_Applications/.ipynb_checkpoints/H-2B_Visa_Programme_2015-checkpoint.ipynb".
The first task is:
7. Work our total pay amount payed to H-2B laborers?
Can you write Python code for it?


I am working on the file "08/H2B_Applications/.ipynb_checkpoints/H-2B_Visa_Programme_2015-checkpoint.ipynb".
The first task is:
9. Most common job title. Graph this.
Can you write Python code for it?

#x = df.groupby('JOB_TITLE')['JOB_TITLE'].value_counts()
df['JOB_TITLE'].value_counts().head(10)

plt.style.use('ggplot')
df['JOB_TITLE'].value_counts(ascending=True).tail(10).plot(kind='barh')
plt.savefig("Top_Jobs.svg")

##Is there an efficient way for Pandas to clean the data? Merge "Landscape Laborer" with "LANDSCAPE LABORER" etc.?


I am working on the file "08/H2B_Applications/.ipynb_checkpoints/H-2B_Visa_Programme_2015-checkpoint.ipynb".
The first task is:
And this table shows that landscape laborers are the ones that are earning the least.
Can you write Python code for it?

lowest_wages_accepted[['BASIC_UNIT_OF_PAY', 'PREVAILING_WAGE', 'EMPLOYER_NAME', 'JOB_TITLE', 'WORKSITE_CITY', 'NBR_WORKERS_REQUESTED', 'NBR_WORKERS_CERTIFIED']]


I am working on the file "08/H2B_Applications/.ipynb_checkpoints/H-2B_Visa_Programme_2015-checkpoint.ipynb".
The first task is:
Approx. ####Count * Mean (Year, Week, Month, Hour(8)(33 Million, Bi-Weekly (180'000)#### 40 million $.
8. Were there any foreign companies hiring foreign workers in the US? If yes, work out averages by nation.
Can you write Python code for it?


I am working on the file "08/H2B_Applications/.ipynb_checkpoints/H-2B_Visa_Programme_2015-checkpoint.ipynb".
The first task is:
Importing the NAIC_Codes from here.
Can you write Python code for it?

NAIC_CODEs = pd.read_excel("6-digit_2012_Code.xls")

NAIC_CODEs.info()

#Changing the NAIC_Codesfrom non-null object into float64

#NAIC_CODEs['NAICS12'] = df['NAIC_CODE'].fillna(0.0).astype(int)

NAIC_CODEs.head()

#And now reimporting the original file.

df = pd.read_excel("H-2B_Disclosure_Data_FY15_Q4.xlsx")

#now in the NAIC_CODE is a Float64 in the cells we want to merge.

df_merged = df.merge(NAIC_CODEs, how = 'left', left_on = 'NAIC_CODE', right_on ='NAICS2012')

#df_merged.info()

df_merged['Industry'].value_counts().head(10)

workers_by_industry = df_merged.groupby('Industry')['NBR_WORKERS_REQUESTED'].sum()
workers_by_industry.sort_values(ascending=True).tail(10).plot(kind='barh', legend='NBR_WORKERS_REQUESTED')
plt.savefig("workers_by_industry.svg")


I am working on the file "08/H2B_Applications/.ipynb_checkpoints/H-2B_Visa_Programme_2015-checkpoint.ipynb".
The first task is:
The majority of the jobs are payed hourly at an average rate of 12.65 $ an hour.
 5. Who earned the least? And where are these people actually doing?
Can you write Python code for it?

worst_wage = df[df['BASIC_UNIT_OF_PAY'] == 'Hour'].sort_values(by='PREVAILING_WAGE', ascending=True).head()


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
1. How many requests did the Office of Foreign Labor Certification (OFLC) receive in 2015?
Can you write Python code for it?


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
10. Which US states have the largest need for unskilled workers? Make a graph of this.
Can you write Python code for it?

#x = df['EMPLOYER_STATE'].value_counts(ascending=False).head(10) * df['NBR_WORKERS_REQUESTED'].sum()
df['EMPLOYER_STATE'].value_counts(ascending=False).head(10).plot(kind='bar')
plt.savefig("semand_in_states.svg")

#x = df['EMPLOYER_STATE'].value_counts(ascending=False).head(10) * df['NBR_WORKERS_REQUESTED'].sum()
df['EMPLOYER_STATE'].value_counts(ascending=True).head(10).plot(kind='bar')
plt.savefig("demand_in_states.svg")

Workers_in_state_count = df.groupby('EMPLOYER_STATE')['NBR_WORKERS_REQUESTED'].sum()
Workers_in_state_count.sort_values(ascending=True).tail(10).plot(kind='barh', legend='NBR_WORKERS_REQUESTED')
plt.savefig("workers_requestet_in_states.svg")


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
11. Which industries had the largest need?
Can you write Python code for it?


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
12. Which companies had the largest need? Compare acceptance/denials of each company.
Can you write Python code for it?

df['EMPLOYER_NAME'].value_counts().head(5)

company_workers_demand = df.groupby('EMPLOYER_NAME')['NBR_WORKERS_REQUESTED'].sum()
company_workers_demand.sort_values(ascending=True).tail(10).plot(kind='barh')
plt.savefig("company_workers_demand.svg")

company_workers_demand = df.groupby('EMPLOYER_NAME')['NBR_WORKERS_CERTIFIED'].sum()
company_workers_demand.sort_values(ascending=True).tail(10).plot(kind='barh')
plt.savefig("company_workers_demand.svg")


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
2. How many jobs did that regard in total? And how many full time positions?
Can you write Python code for it?


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
3. How many jobs did the ETA National Processing Center actually certify?
Can you write Python code for it?


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
6. What was the most common unit of pay (daily, weekly, monthly)?
Can you write Python code for it?


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
9. Most common job title. Graph this.
Can you write Python code for it?


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
And this table shows that landscape laborers are the ones that are earning the least.
Can you write Python code for it?


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
Approx. ####Count * Mean (Year, Week, Month, Hour(8)(33 Million, Bi-Weekly (180'000)#### 40 million $.
8. Were there any foreign companies hiring foreign workers in the US? If yes, work out averages by nation.
Can you write Python code for it?


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
BONUS Looking into Silver Bay Seafoods and UK International Soccer Campus.
Silver Bay's claim: [Silver Bays's "...primary strength is in its combination of having a state of the art processing plant and favorable logistics to support its operations; competent management and key personnel; an established fish buying system; and ownership by fishermen who represent over 80% of the committed fishing effort."] How much does the company pay it's workers on average?
Can you write Python code for it?

SILVER_BAY_SEAFOODS = df[df['EMPLOYER_NAME'] == 'SILVER BAY SEAFOODS, LLC']

SILVER_BAY_SEAFOODS[['JOB_TITLE', 'PREVAILING_WAGE', 'HOURLY_WORK_SCHEDULE_AM', 'HOURLY_WORK_SCHEDULE_PM', 'OVERTIME_RATE_FROM', 'OVERTIME_RATE_TO', 'NATURE_OF_TEMPORARY_NEED', 'NBR_WORKERS_REQUESTED', 'NBR_WORKERS_CERTIFIED']]

SOCCER_CAMPS = df[df['EMPLOYER_NAME'] == 'UK International Soccer Camps']
SOCCER_CAMPS[['JOB_TITLE', 'PREVAILING_WAGE', 'HOURLY_WORK_SCHEDULE_AM', 'HOURLY_WORK_SCHEDULE_PM', 'OVERTIME_RATE_FROM', 'OVERTIME_RATE_TO', 'NATURE_OF_TEMPORARY_NEED', 'NBR_WORKERS_REQUESTED', 'NBR_WORKERS_CERTIFIED']]

df_unfaelle.groupby('Jahr')[(df_unfaelle['Lichtverhältnis UAP'] == 622)]


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
Importing the NAIC_Codes from here.
Can you write Python code for it?


I am working on the file "08/H2B_Applications/H-2B_Visa_Programme_2015.ipynb".
The first task is:
The majority of the jobs are payed hourly at an average rate of 12.65 $ an hour.
 5. Who earned the least? And where are these people actually doing?
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
.. And for the kind of tenure
Can you write Python code for it?

slums['percentageoftenure_patta'] = slums['tenure_patta']/slums['households']*100
slums['percentageoftenure_pc'] = slums['tenure_possession_certificate']/slums['households']*100
slums['percentageoftenure_private'] = slums['tenure_private_land']/slums['households']*100
slums['percentageoftenure_public'] = slums['tenure_public']/slums['households']*100
slums['percentageoftenure_renters'] = slums['tenure_rented']/slums['households']*100
slums['percentageoftenure_other'] = slums['tenure_others']/slums['households']*100


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
15. What percentage of Households have access to Possession Certificates?
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
16. What percentage of Households are on Private Land?
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
17. What percentage of Households are on Public Land?
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
19. What percentage of Households don't fit in any of the other categories?
Can you write Python code for it?

f=slums['percentageoftenure_other'].mean()

plt.style.use('ggplot')
series=pd.Series([a,b,c,d,e,f], index=['Pattas', 'PC', 'Private Land','Public Land','Renters','Others'], name='Tenuree Type')
series.plot.pie(figsize=(6, 6))


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
21. What percentage of Households have moved in the last one to three years?
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
22. What percentage of Households have moved in the last three to five years?
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
23. What percentage of Households have moved more than five years ago?
Can you write Python code for it?

d=slums['percentageoftenure_morethan5'].mean()

plt.style.use('ggplot')
series=pd.Series([a,b,c,d], index=['Zero to One Years', 'One to Three Years', 'Three to Five Years','More than Five Years'])
series.plot.pie(figsize=(6, 6))


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
25. What is the average monthly expenditure of a slum household? (in rupees, 70 rupees make a dollar)
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
3. Which circles have the maximum number of slums?
Circle here refers to a Municipal Subdivision. The city is divided into 18 circles, with each circle being further subdivided into wards. The city has 150 of these wards.
Can you write Python code for it?

circle_count=slums['circle_number'].value_counts()
circle_count

plt.style.use('ggplot')
circle_count.plot(kind='bar',x='Circle Number',y='Number of Slums',legend=False, figsize=(10,10))


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
6. What percentage of households in the slum are BPL(Below Poverty Level)?
As argued above, there are valid concerns that the slum might not be a great proxy for urban powerty. But if the percentage of households that are Below Poverty Line is high enough, then the slum can be considered as a proxy for urban poverty.
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
8. Which slums have the highest percentage of households that belong to the SC Caste?
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
A quarter of the city's population approximately lives in slums.
2. What is the average number of households in a slum? What is the average family size?
Can you write Python code for it?

print("The total number of slum households is",slums['households'].sum())
print("")
print("The average number of households in a slum is",slums['households'].mean())
print("")
print("The average family size is", (slums['population'].sum()/slums['households'].sum()))


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
Because the data is faulty, there are slums whose ward numbers is empty, ie 0.The remainder of the list are the ward numbers with maximum slums and on the right are the number of slums in them. That makes ward 108 have the maximum number of slums. On introspecting on a map(in CartoDB) - one is curious to know why is it that this ward has a highly fragmented number of slums, often not more than a couple of houses.
5. What is the total literacy level in the slums?
We count the total number of literates in each slum and compare it with the total population.
Can you write Python code for it?

print("The total percentage of literacy in slums is",(slums['literacy_literates'].sum())/(slums['population'].sum())*100)


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
Circle 4 - which is the old city quarter of Hyderabad has the maximum number of slums in terms of numbers.
4. Which wards have the maximum number of slums?
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
INCOME, EXPENDITURE, DEBT
24. What is the average monthly income of a slum household? (in rupees, 70 rupees make a dollar)
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
Kaccha houses.
LAND TENURE DETAILS
14. What percentage of Households have access to Pattas?
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
Now let's look at the columns in the database.
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
Now we repeat this for the Caste Parameters.
Can you write Python code for it?

slums['percentageofgeneral'] = slums['caste_general']/slums['households']*100
slums['percentageofminority']= slums['minority']/slums['households']*100
slums['percentageofobc'] = slums['caste_obc']/slums['households']*100
slums['percentageofsc'] = slums['caste_sc']/slums['households']*100
slums['percentageofst'] = slums['caste_st']/slums['households']*100


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
Now, let us see how the slums dataframe looks like
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
On closer introspection, it is found that the other datasets are per household and not population wise like tha literacy percentage. So we do the addition of the following columns into the dataframe. So we repeat the above calculation for percentage of households below poverty line.
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
So there are the following things in the dataset.
* Name of the Slums

The Municipal Circle and Ward Number

The number of Households and Population slumwise.

The number of Households that are Below Poverty Line (BPL) - It means there are people in slums who are not poor as defined by the government's definition. This becomes crucial because - it is often assumed that the slum is a proxy for urban poverty. Gautam Bhan and Arindam Jana's paper argues otherwise (http://epw.yodasoft.com/journal/2015/22/review-urban-affairs-review-issues/reading-spatial-inequality-urban-india.html)

Caste Parameters : In Indian cities, Caste is an important index. The households that are "General", "Scheduled Caste", "Scheduled Tribe" or "Other Backward Castes" are all counted and have their own columns.

Tenure Parameters: Because the dataset is primarily about eliminating slums with the idea of providing Government built houses, the land tenure status of the slum households is essential to know. They are Patta, Possession Certificate, Slums on Private Land, Slums on Public Land, Households that are renters and Other tenures that do not fit under this framework.

Structure of the House : This pertains to what kind of a house the slum dweller lives on. Pucca means a stable house made out of concrete, Semipucca could mean a not so stable house and Kuccha house means a house made out of mud/wood/etc.

Average Monthly Income, Expenditure and Debt.

The number of years have the dwellers stayed in the slums

Number of Female Headed Households.

1. What are the total number of slums in Hyderabad? What is the number of people who live in slums? How much percentage of the city's population lives in slums?
Can you write Python code for it?

totalpopulation=slums['population'].sum()
print("The total number of people who live in slums in Hyderabad are",int(totalpopulation))
print("")
populationofhyderabad=6731790 #According to the 2011 census.
percentageofpopulation=(totalpopulation/populationofhyderabad)*100
print("The percentage of people who live in slums is",(percentageofpopulation))


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
That is high enough for us to consider the slum as a decent proxy for urban poverty in Hyderabad.
The remaining data in its absolute number forms make it impossible for us to compare slums among each other. It would be useful to convert the caste, tenure, age, structures into percentages, making it easy to compare.
Now, we create new columns in the dataframe using the existing columns and check.
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
WHEN HAVE PEOPLE MOVED IN
20. What percentage of Households have moved in the last one year?
Can you write Python code for it?


I am working on the file "08/Homework 8 - Dataset - Devulapalli.ipynb".
The first task is:
We now have the data with all the calculations necessary made for analysis and now we begin.
CASTE PARAMETERS
7. Which slums have the highest percentage of households that belong to the General Caste?
Can you write Python code for it?


I am working on the file "08/Homework8-passengers.ipynb".
The first task is:
Graphics
Can you write Python code for it?

import matplotlib.pyplot as plt

%matplotlib inline

plt.style.use('ggplot')

standard = df[(df['DWV'] > 300) & (df['DWV'] < 2300) ]

standard.plot(kind='scatter', x='DWV', y='DTV')
print("These are the stations in Q2 and Q3 and their average daily passengers during the full week vs. the work days:")

plt.style.use('ggplot')
least_frequented = df.sort_values(by='DWV').head(20)
least_frequented.plot(kind='barh', x='Station', y='DTV').invert_yaxis()
print("These are the 20 least frequented stations, in average daily passengers:")

q1_freq = df[df['DWV'] <= 340]
q2_freq = df[(df['DWV'] <= 915) & (df['DWV'] > 340)]
q3_freq = df[(df['DWV'] <= 2700) & (df['DWV'] > 915)]

plt.scatter(y=q1_freq["DWV"], x=q1_freq["DTV"], c='c', alpha=0.75, marker='1')
plt.scatter(y=q2_freq["DWV"], x=q2_freq["DTV"], c='y', alpha=0.75, marker='2')
plt.scatter(y=q3_freq["DWV"], x=q3_freq["DTV"], c='m', alpha=0.75, marker='3')

print("Q1, Q2 and Q3 of average daily circulation; x axis = DTV, y axis = DWV")

plt.xlim(-15,2500)
plt.ylim(-30,2800)


I am working on the file "08/Homework_8_Emelike.ipynb".
The first task is:
1) Open your dataset up using pandas in a Jupyter notebook
Can you write Python code for it?

df = pd.read_csv('Mother Jones US Mass Shootings 1982-2016 - US mass shootings.csv')


I am working on the file "08/Homework_8_Emelike.ipynb".
The first task is:
10) In which kind of venue do most mass shootings occur?
Can you write Python code for it?

df['Venue'] = df['Venue'].str.replace('Other\n', 'Other')
df['Venue'] = df['Venue'].str.replace('\nWorkplace', 'Workplace')
df['Venue'].value_counts()


I am working on the file "08/Homework_8_Emelike.ipynb".
The first task is:
2) Do a .head() to get a feel for your data
Can you write Python code for it?