I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
The other dataframe we dowloaded (using code NVCA/VENTURE_3_09C) contains quarterly data on total investment by venture capital firms in the US, broken down by the stage of the project.
The column names here are ok, so we don't need to change anything.
So that we have the data easily acessible for later on, let's store these two variables in individual dataframes:
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
Time deltas work the same way with datetime objects as they did with date objects.
We can see how many seconds until Spencer turns 30:
Can you write Python code for it?

# NOTE: we can only do arithmetic between many date objects or datetime obejcts
#       we cannot add or subtract a datetime to/from a date. So, we need to
#       re-create spencer_bday as a datetime object.
# NOTE: The timedelta object is already compatible with date and datetime objects

spencer_bday_time = dt.datetime(1989, 4, 25, 16, 33, 5)
seconds_to_30 = (spencer_bday_time + thirty_years - now).seconds

print("Spencer will be 30 in {} seconds".format(seconds_to_30))


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
To resolve the second issue we will do two-steps:

resample the ffr data to a monthly frequency
resample the vc data to a monthly frequency by padding. This is called upsampling because we are going from a lower frequency (quarterly) to a higher one (monthly)
Can you write Python code for it?

ffr_recentM = ffr_recent.resample("M").first()
vc_M = vc.resample("M").pad()

vc_M.head()


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
We can get the number of days until new years eve by looking at until_nye.days
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
What happened to the first column?
Can you write Python code for it?

mix = quandl.get(['NSE/OIL.1', 'WIKI/AAPL.4'])
mix.plot(subplots=True)

df_info(dfs[1])


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
datetime
Being able to work with dates and the difference between dates is very useful, but sometimes we need to also think about times. To do that, we will look to the dt.datetime module.
We can get the current date and time using dt.datetime.now():
Can you write Python code for it?

now = dt.datetime.now()
print("type of now:", type(now))
now


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
more than you need: I want to point out that when you use the first or last methods to perform the aggregations, there are two dates involved: (1) the date the resultant index will have and (2) the date used to fill in the data at that date. The first date (one on the index) will be assigned based on the string you pass to the resample method. The second date (the one for extracting data from the original dataframe) will be determined based on the method used to do the aggregation. first will extract the first data point from that subset and last will extract the last.
Let's see some examples:
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
resampling
Now suppose that instead of daily data, we wanted our federal funds data at a monthly frequency.
To do this we use the resample method on our DataFrame
Can you write Python code for it?

# MS means "month start"
ffrM_resample = ffr.resample("MS")
type(ffrM_resample)


I am working on the file "Code/notebooks/bootcamp_timeseries_update.ipynb".
The first task is:
strftime
Once we have date and time information, a very common thing to do is to print out a formatted version of that date. For example, suppose we wanted to print out a string in the format YYYY-MM-DD. To do this we use the strftime method. Here's an example
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_web_apis.ipynb".
The first task is:
Exercise (10 min) WARNING: this is a long exercise, but should make you use tools from almost every lecture of the last 6 weeks.
Our want is:

A DataFrame with one column for each of those 5 variables
The index should be the time period and should have type DatetimeIndex
The dtype for all columns should be float64

Here's an outline of how I would do this:

Remove all rows where Series code isn't one of our 5 variables (now named GDP, C, G, etc.)
drop all columns we don't need
Convert the TimePeriod column to a datetime (HINT: use pd.to_datetime)
convert the DataValue column to have the correct dtype (HINT: you'll need to use the .str methods here)
At this point you have 3 columns, all with the right dtype. Now use some combination of set_index and unstack to get the correct row and column labels (HINT: You might have ended up with 2 levels on your column index (I did) -- drop the one for DataValue if necessary)

Test out how well this went by plotting the DataFarme
Open Data Network API
The Open Data Network is a collection of cities, states, and Federal Governmental agencies that have all open accessed their data using the same tools. If you follow the link to the Open Data Network, there is a list of all cities that participate at the bottom. It includes New York City, Chicago, Boston, Houston, and many more.
The tool all of these cities are using to open source their data is called Socrata. One of the benefits of using the same tool is that it leads to being able to access various datasets using the same API.
The general API documentation can be found here. Let's open this up and see whether we can extract some of the important pieces of information that we'd like. We need to find two things:

A "root url" that we put at the beginning of all of our requests
The set of parameters that we want to define for any request (information like what dataset, how many observations, or what time frame).

This API has some nice features that you won't necessarily get on other APIs. One of these...
Can you write Python code for it?

chi_apie = "https://data.cityofchicago.org/"
chi_crime_url = chi_apie + "resource/6zsd-86xi.json?$limit=25000"
chi_df = pd.read_json(chi_crime_url)

chi_df.head()[["arrest", "case_number", "community_area", "date"]]


I am working on the file "Code/notebooks/bootcamp_web_apis.ipynb".
The first task is:
Exercise (2 min): Experiment with my_func to make sure you understand how it works. You might try these things out:

Why doesn't my_func(1) work?
What is the type of x in x = my_func(a=1, b=2)?
What is the type of and len of x in x = my_func()?

Let's test out our bea_request function by calling the GetDataSetList method.
First, we need to check the methods page of the documentation to make sure we don't need any additional parameters. Looks like this one doesn't. Let's call it and see what we get
Can you write Python code for it?

datasets_raw = bea_request("GetDataSetList")
type(datasets_raw)

# did the request succeed?
datasets_raw.ok

# status code 200 means success!
datasets_raw.status_code

datasets_raw.content


I am working on the file "Code/notebooks/bootcamp_web_apis.ipynb".
The first task is:
Exercise: Find the API endpoint for Boston crime (use the Crime Incident Reports July 2012-August 2015 data).
Exercise: Read in the first 50 observations of the Boston crime dataset into a dataframe named bos_df
We can now look at what types everything in these two datasets are and look at what information is contained in them.
Can you write Python code for it?


I am working on the file "Code/notebooks/bootcamp_web_apis.ipynb".
The first task is:
Notice that this dict has one item. The key is BEAAPI. The value is another dict. Let's take a look inside this one
Can you write Python code for it?

datasets_dict = datasets_raw_dict["BEAAPI"]
print("length of datasets_dict:", len(datasets_dict))
datasets_dict


I am working on the file "Code/notebooks/bootcamp_web_apis.ipynb".
The first task is:
Plot Chicago crime over time
Recall, we only have the first 25000 elements of the dataset, so the results are likely to be nonsense. We do it anyways because it gives us a chance to use the timeseries tools we talked about previously.
Can you write Python code for it?

chi_df = chi_df.set_index("date")

cases_per_month = chi_df.resample("M").count()["case_number"]

cases_per_month.plot()


I am working on the file "Code/notebooks/bootcamp_web_apis.ipynb".
The first task is:
The ParameterName column above tells us the name of all additional parameters we can send to GetData.
The ParameterIsRequiredFlag has a 1 if that parameter is required and a 0 if it is optional
Finally, the ParameterDataType tells us what type the value of each parameter should be.
I did a of digging and found that the GDP data we are after lives in table 6. Let's get quarterly data for 1990 to 2016
Can you write Python code for it?

gdp_data = bea_request("GetData", DataSetName="NIPA",
                      TableId=6,
                      Frequency="Q",
                      Year=list(range(1990, 2017)))

# check to make sure we have a 200, meaning success
gdp_data.status_code

# extract the results and read into a DataFrame
gdp = pd.DataFrame(gdp_data.json()["BEAAPI"]["Results"]["Data"])
print("The shape of gdp is", gdp.shape)
gdp.head()


I am working on the file "Code/notebooks/bootcamp_web_apis.ipynb".
The first task is:
The important columns for us are going to be DataValue,  SeriesCode, and TimePeriod. I did a bit more digging and found that the series codes map into our variables as follows
Can you write Python code for it?


I am working on the file "Code2/Python/Annotation_Notebook.ipynb".
The first task is:
Functions
Can you write Python code for it?

def mask_l2ball(BF):
    a = np.zeros((2 * BF + 1, 2 * BF + 1, 2 * BF + 1))
    for i in range(a.shape[0]):
        for j in range(a.shape[1]):
            for k in range(a.shape[2]):
                if LA.norm(np.asarray([k - BF, j - BF, i - BF])) <= BF:
                    a[k,j,i] = 1

    out = np.asarray(a, dtype = np.uint8)
    return(out)

def main(base_path, base_fname, locations, dim, buf = [5,5,5], inftyball = False, l2ball = False):
    #parser = argparse.ArgumentParser('Post rectangular annotations stack to the boss')
    #parser.add_argument('locations', type=str, help='centroids in x y z csv file')
    #parser.add_argument('base_path', type=str, help='Directory where image stacks are to be saved (e.g. "/data/images/"')
    #parser.add_argument('base_filename', type=str, help='Base filename with z values appended "fname_z<p:4>"')
    #parser.add_argument('--buf', type=int, nargs=3, help='buffer in x y z', default=[5,5,5])
    #parser.add_argument('--dim', type=int, nargs=3, help='dimensions in x y z')
    #parser.add_argument('--res', type=int, default=0, help='Resolution at which we post data.')
    #parser.add_argument('--inftyball', action='store', type=bool, default=False, help='bool to use infinity balls.')
    #parser.add_argument('--l2ball', action='store', type=bool, default=False, help='bool to use l2 balls, only uses buf[0].')

    #args = parser.parse_args()

    base_path
    base_fname

    dimx = dim[0]
    dimy = dim[1]
    dimz = dim[2]
    loc = locations

    L = genfromtxt(loc,  delimiter=',', skip_header = 1, dtype='int32').tolist()

    BFx = buf[0]
    BFy = buf[1]
    BFz = buf[2]

    x_rng = [[x[0] - BFx, x[0] + BFx + 1] for x in L]
    y_rng = [[y[1] - BFy, y[1] + BFy + 1] for y in L]
    z_rng = [[z[2] - BFz, z[2] + BFz + 1] for z in L]

    inftyones = np.ones((2 * BFz + 1, 2 * BFy + 1, 2 * BFx + 1), dtype = np.uint8)

    if inftyball:
        anno = np.asarray(255*inftyones, dtype = np.uint8)
        ## NUMPY array must be in Z Y X...


I am working on the file "Code2/Python/Annotation_Notebook.ipynb".
The first task is:
Small test example
Can you write Python code for it?

import urllib.request

base_path = "/tmp/"
base_fname = "testEx2R18C1"
locations = "Ex2R18C1_BufferedPM5.csv"
buf = [5,5,5] # X, Y, Z order
dim = [2101, 3223, 69] # X, Y, Z order
res = 0

#url = 'http://cis.jhu.edu/~jesse/data/Ex2R18C1_buff5seed317_samp1e4.csv'
#response = urllib.request.urlopen(url)
#data = response.read()      # a `bytes` object
#locations = response #.decode('utf-8') # a `str`; this step

#main(base_path, base_fname, locations, dim, buf, inftyball = True, l2ball = False)
main(base_path, base_fname, locations, dim, buf, inftyball = False, l2ball = True)


I am working on the file "Code_Report.ipynb".
The first task is:
As for the optimization, we employed vectorization to avoid the use of triple for-loops under the update section of the Baum-Welch algorithm. We used broadcasting with numpy.newaxis to implement Baum-Welch algorithm much faster. As we can see from Benchmark part in the report, under class HMM we have 2 functions for Baum-Welch algorithm called Baum_Welch and Baum_Welch_fast. In Baum_Welch_fast, vectorization is applied when calculating
$\xi$ while in Baum_Welch, we use a for loop. Notice in Baum_Welch, all other parts are implemented with vectorization. This is just an example how vectorization greatly improve the speed. Notice that the run time for vectorized Baum-Welch algorithm is 2.43 s per loop (with scaling) and 1 s per loop (without scaling) compared to 4.01 s per loop (with scaling) and 261 s per loop (without scaling). Other functions are implemented with vectorization as well. Vectorization greatly improves our time performance.
Simulations
Effect of chain length
Can you write Python code for it?


I am working on the file "Code_Report.ipynb".
The first task is:
Benchmark of vectorization
Can you write Python code for it?

pi=np.array([.3,.3,.4])
A=np.array([[.2,.3,.5],[.1,.5,.4],[.6,.1,.3]])
B=np.array([[0.1,0.5,0.4],[0.2,0.4,0.4],[0.3,0.6,0.1]])
states,sequence=HMM.sim_HMM(A,B,pi,100)

%timeit HMM.Baum_Welch(A,B,pi,sequence,1000,0,scale=True)
%timeit HMM.hmm_unoptimized.Baum_Welch(A,B,pi,sequence,1000,0,scale=True)
%timeit HMM.Baum_Welch(A,B,pi,sequence,1000,0,scale=False)
%timeit HMM.hmm_unoptimized.Baum_Welch(A,B,pi,sequence,1000,0,scale=False)


I am working on the file "Code_Report.ipynb".
The first task is:
Comparative Analysis
Can you write Python code for it?


I am working on the file "Code_Report.ipynb".
The first task is:
Comparing Viterbi decoding
Can you write Python code for it?

mod=hmm.MultinomialHMM(n_components=3)
mod.startprob_=pi
mod.transmat_=A
mod.emissionprob_=B
res_1=mod.decode(np.array(sequence).reshape([100,1]))[1]

res_2=HMM.Viterbi(A,B,pi,sequence)

np.array_equal(res_1,res_2)

%timeit -n100 mod.decode(np.array(sequence).reshape([100,1]))
%timeit -n100 HMM.Viterbi(A,B,pi,sequence)


I am working on the file "Code_Report.ipynb".
The first task is:
From this part, we can see that the choice of initial values are greatly important. Because Baum-Welch algorithm does not guarantee global maximum, a bad choice of initial values will make Baum-Welch Algorithm to be trapped in a local maximum. Moreover, our experiments show that the initial values for emission matrix $B$ are more important by comparing initial values 3 and 4. The initial parameters represent your belief.
Effect of number of iteration in Baum-Welch Algorithm
Can you write Python code for it?

############INITIAL VALUES 1###############
A_init=np.array([[0.2,0.6,0.2],[0.25,0.5,0.25],[0.6,0.2,0.2]])
B_init=np.array([[0.05,0.1,0.15,0.7],[0.4,0.4,0.1,0.1],[0.6,0.2,0.2,0.2]])
pi_init=np.array([0.3,0.3,0.4])
n_iter=[1,5,10,25,50,100,500]
acc=np.zeros([k,len(n_iter)])
k=30
for j in range(k):
    states,sequence=HMM.sim_HMM(A,B,pi,100)
    t=0
    for i in n_iter:
        Ahat,Bhat,pihat=HMM.Baum_Welch(A_init,B_init,pi_init,
                                       sequence,i,0,False)
        seq_hat=HMM.Viterbi(Ahat,Bhat,pihat,sequence)
        acc[j,t]=np.mean(seq_hat==states)
        t+=1
plt.plot(n_iter,np.mean(acc,axis=0))


I am working on the file "Code_Report.ipynb".
The first task is:
In this initial condition, we can see one feature of Baum-Welch Algorithm: Baum-Welch Algorithm tends to overfit the data, which is $P(Y|\theta_{final})>P(Y|\theta_{true})$.
Can you write Python code for it?

############INITIAL VALUES 2###############
A_init=np.array([[0.5,0.25,0.25],[0.1,0.4,0.5],[0.25,0.1,0.65]])
B_init=np.array([[0.3,0.4,0.2,0.1],[0.1,0.5,0.2,0.2],[0.1,0.1,0.4,0.4]])
pi_init=np.array([0.5,0.2,0.3])
n_iter=[1,5,10,25,50,100,500]
acc=np.zeros([k,len(n_iter)])
k=30
for j in range(k):
    states,sequence=HMM.sim_HMM(A,B,pi,100)
    t=0
    for i in n_iter:
        Ahat,Bhat,pihat=HMM.Baum_Welch(A_init,B_init,pi_init,
                                       sequence,i,0,False)
        seq_hat=HMM.Viterbi(Ahat,Bhat,pihat,sequence)
        acc[j,t]=np.mean(seq_hat==states)
        t+=1
plt.plot(n_iter,np.mean(acc,axis=0))

############INITIAL VALUES 3###############
A_init=np.array([[0.2,0.6,0.2],[0.25,0.5,0.25],[0.6,0.2,0.2]])
B_init=np.array([[0.3,0.4,0.2,0.1],[0.1,0.5,0.2,0.2],[0.1,0.1,0.4,0.4]])
pi_init=np.array([0.5,0.2,0.3])
n_iter=[1,5,10,25,50,100,500]
acc=np.zeros([k,len(n_iter)])
k=30
for j in range(k):
    states,sequence=HMM.sim_HMM(A,B,pi,100)
    t=0
    for i in n_iter:
        Ahat,Bhat,pihat=HMM.Baum_Welch(A_init,B_init,pi_init,
                                       sequence,i,0,False)
        seq_hat=HMM.Viterbi(Ahat,Bhat,pihat,sequence)
        acc[j,t]=np.mean(seq_hat==states)
        t+=1
plt.plot(n_iter,np.mean(acc,axis=0))

############INITIAL VALUES 4###############
A_init=np.array([[0.5,0.25,0.25],[0.1,0.4,0.5],[0.25,0.1,0.65]])
B_init=np.array([[0.05,0.1,0.15,0.7],[0.4,0.4,0.1,0.1],[0.6,0.2,0.2,0.2]])
pi_init=np.array([0.5,0.2,0.3])
n_iter=[1,5,10,25,50,100,500]
acc=np.zeros([k,len(n_iter)])
k=30
for j in range(k):
    states,sequence=HMM.sim_HMM(A,B,pi,100)
    t=0
    for i in n_iter:
        Ahat,Bhat,pihat=HMM.Baum_Welch(A_init,B_init,pi_init,
                                       sequence,i,0,False)
        seq_hat=HMM.Viterbi(Ahat,Bhat,pihat,sequence)
        acc[j,t]=np.mean(seq_hat==states)
        t+=1
plt.plot(n_iter,np.mean(acc,axis=0))


I am working on the file "Code_instructions.ipynb".
The first task is:
Also, we have to define the boundary functions that we will use to apply the boundary conditions. In this case an armonic wave for Dirichlet and his derivate for Neumann
Can you write Python code for it?

#Funciones de dirichlet y neumann
def dirichlet_fun(x, n, domain_index, result):
        result[0] = 1.*np.exp(1j*k*x[0])
def neumann_fun(x, n, domain_index, result):
        result[0] = 1.*1j*k*n[0]*np.exp(1j*k*x[0])


I am working on the file "Code_instructions.ipynb".
The first task is:
Below are showed the form to assembly all the operators in the big block:
Can you write Python code for it?

#Diagonal
blocked[0,0] = op_m[0,0]
blocked[0,1] = op_m[0,1]
blocked[1,0] = op_m[1,0]
blocked[1,1] = op_m[1,1]
blocked[2,2] = op_0[0,0]
blocked[2,3] = op_0[0,1]
blocked[3,2] = op_0[1,0]
blocked[3,3] = op_0[1,1]
blocked[4,4] = op_1[0,0]
blocked[4,5] = op_1[0,1]
blocked[5,4] = op_1[1,0]
blocked[5,5] = op_1[1,1]

#Contribucion hilos-matriz
blocked[0,2] = DLP_0_m
blocked[0,3] = -SLP_0_m
blocked[1,2] = -HYP_0_m
blocked[1,3] = -ADLP_0_m
blocked[0,4] = DLP_1_m
blocked[0,5] = -SLP_1_m
blocked[1,4] = -HYP_1_m
blocked[1,5] = -ADLP_1_m

#Contribucion hilos-hilos
blocked[2,4] = DLP_1_0
blocked[2,5] = -SLP_1_0
blocked[3,4] = -HYP_1_0
blocked[3,5] = -ADLP_1_0

#Contribucion matriz-hilos
blocked[2,0] = -DLP_m_0
blocked[2,1] = SLP_m_0
blocked[3,0] = HYP_m_0
blocked[3,1] = ADLP_m_0

#Contribucion hilos-hilos
blocked[4,2] = DLP_0_1
blocked[4,3] = -SLP_0_1
blocked[5,2] = -HYP_0_1
blocked[5,3] = -ADLP_0_1

#Contribucion matriz-hilos
blocked[4,0] = -DLP_m_1
blocked[4,1] = SLP_m_1
blocked[5,0] = HYP_m_1
blocked[5,1] = ADLP_m_1


I am working on the file "Code_instructions.ipynb".
The first task is:
Finally we plot the convergence and export it
Can you write Python code for it?

import matplotlib
matplotlib.use("Agg")
from matplotlib import pyplot
from matplotlib import rcParams
rcParams["font.family"] = "serif"
rcParams["font.size"] = 20
pyplot.figure(figsize = (15,10))
pyplot.title("Convergence")
pyplot.plot(array_it, array_frame, lw=2)
pyplot.xlabel("iteration")
pyplot.ylabel("residual")
pyplot.grid()
pyplot.savefig("Convergence.pdf")


I am working on the file "Code_instructions.ipynb".
The first task is:
In order to obtain the spaces created with the multitrace opertaor it's posible to do the following:
Can you write Python code for it?

#Espacios
dirichlet_space_m = Ai_m[0,0].domain
neumann_space_m = Ai_m[0,1].domain
dirichlet_space_0 = Ai_0[0,0].domain
neumann_space_0 = Ai_0[0,1].domain
dirichlet_space_1 = Ai_1[0,0].domain
neumann_space_1 = Ai_1[0,1].domain


I am working on the file "Code_instructions.ipynb".
The first task is:
In the following lines are the codes to import the meshes that we will use
Can you write Python code for it?

#Importando mallas
matriz = bempp.api.import_grid('/home/milan/matriz_12x12x300_E16772.msh')
grid_0 = bempp.api.import_grid('/home/milan/PH1_a5_l10_E5550_D2.msh')
grid_1 = bempp.api.import_grid('/home/milan/PH2_a5_l10_E5550_D2.msh')


I am working on the file "Code_instructions.ipynb".
The first task is:
The definition of boundary conditions, the discretization of the operators and the discretization of right side are:
Can you write Python code for it?

#Condiciones de borde
dirichlet_grid_fun_m = bempp.api.GridFunction(dirichlet_space_m, fun=dirichlet_fun)
neumann_grid_fun_m = bempp.api.GridFunction(neumann_space_m, fun=neumann_fun)

#Discretizacion lado izquierdo
blocked_discretizado = blocked.strong_form()

#Discretizacion lado derecho
rhs = np.concatenate([dirichlet_grid_fun_m.coefficients, neumann_grid_fun_m.coefficients,np.zeros(dirichlet_space_0.global_dof_count), np.zeros(neumann_space_0.global_dof_count), np.zeros(dirichlet_space_1.global_dof_count), np.zeros(neumann_space_1.global_dof_count)])


I am working on the file "Code_instructions.ipynb".
The first task is:
The first subinedx corresponds to the domain space, the second one to the range space. Now is time to create the big block that will have all the operators together, in this case the size is 6X6
Can you write Python code for it?

#Matriz de operadores
blocked = bempp.api.BlockedOperator(6,6)


I am working on the file "Code_instructions.ipynb".
The first task is:
To make the complete diagonal of the main matrix showed at beggining is necessary to define the identity operators:
Can you write Python code for it?

#Operadores identidad
ident_m = bempp.api.operators.boundary.sparse.identity(neumann_space_m, neumann_space_m, neumann_space_m)
ident_0 = bempp.api.operators.boundary.sparse.identity(neumann_space_0, neumann_space_0, neumann_space_0)
ident_1 = bempp.api.operators.boundary.sparse.identity(neumann_space_1, neumann_space_1, neumann_space_1)


I am working on the file "CoderLaVille.ipynb".
The first task is:
Cartographie des équipements et des acteurs du numérique
Can you write Python code for it?


I am working on the file "Codigo/Documentación.ipynb".
The first task is:
¿Cómo funciona el programa?
Los códigos realizados para este proyecto de aula estan implementados en python y utilizan los siguientes paquetes:
 + Requests
 + Beautifulsoup4
 + Webbrowser
 + numpy
 + Ipython
descarga.py
Este archivo es el encargado de la extracción web de la base de datos que se descargará como archivo de texto plano con el nombre de tzr.csv.
Este código ingresa a la página http://montcoalert.org/gettingdata/ que es nuestra fuente de datos y busca la url del archivo tzr.csv que esta nos facilita y lo descarga en el directorio Codigo.
consulta.py
Este archivo contiene dos funciones:
+ Datos_necesarios():Carga el archivo de texto a una matriz y luego selecciona las columnas de los datos con mas relevancia en este caso y lo carga a la matriz  donde se realiza la consulta.
+ Consulta():Solicita al usuario que ingrese el tipo de dato y el dato de búsqueda,luego llama a la función Datos_necesarios() que se encarga de buscar las coincidencias del dato ingresado en la matriz. Inmediatamente después genera una tabla markdown con los resultados de la búsqueda.
Estadistica-Analisis de Datos
Despúes de realizar la instalacion,inicializacion indicadas anteriormente(si no esta en el directorio Codigo dirigase a el) y ejecute GenArEst.py como se muestra a continuación.
Can you write Python code for it?


I am working on the file "Coding/Catalog Analysis.ipynb".
The first task is:
Plot 01: Redshift distribution of our sample
Can you write Python code for it?

redshift = my_dictionary['Z_HELIO'].astype(float)

sns.boxplot(y=redshift)
plt.show()

sns.boxplot(y=redshift, showfliers=False)
plt.show()


I am working on the file "Coding/Catalog Analysis.ipynb".
The first task is:
Plot 07: Absolute Magnitude r band x Redshift
Can you write Python code for it?

plt.plot(redshift[idx_nospurious], mag_abs_r[idx_nospurious], 'o', color='#fdae6b', alpha=0.7, label='All objects')
plt.plot(redshift[index_uvup], mag_abs_r[index_uvup], 'o', color='#e6550d', alpha=0.7, label='UV Upturn')
plt.xlabel("Redshift", fontsize=15)
plt.ylabel("M$_{r}$", fontsize=15)
plt.legend(loc='best', numpoints=1, fontsize=20, frameon=False)
plt.savefig('./Figs/mag_redshift.png', dpi=100)
plt.show()

plt.hexbin(redshift[idx_nospurious], mag_abs_r[idx_nospurious], gridsize=100, bins='log', label='All objects')
plt.xlabel("Redshift", fontsize=15)
plt.ylabel("M$_{r}$", fontsize=15)
plt.legend(loc=4, numpoints=1, fontsize=20, frameon=False)
plt.savefig('./Figs/mag_redshift_hexbin.png', dpi=100)
plt.show()

plt.hexbin(redshift[index_uvup], mag_abs_r[index_uvup], gridsize=20, bins='log', label='UV Upturn')
plt.xlabel("Redshift", fontsize=15)
plt.ylabel("M$_{r}$", fontsize=15)
plt.legend(loc=4, fontsize=20, frameon=False)
plt.savefig('./Figs/mag_redshift_uv_hexbin.png', dpi=100)
plt.show()


I am working on the file "Coding/Catalog Analysis.ipynb".
The first task is:
Plot 07: Some distributions
Can you write Python code for it?

plt.hist((g_band - r_band)[idx_nospurious], bins=np.arange((g_band - r_band)[idx_nospurious].min(), (g_band - r_band)[idx_nospurious].max(), 0.15), color='#e6550d',
         log=True)
plt.xlabel("g-r", fontsize=15)
plt.ylabel("Frequency", fontsize=15)
plt.tick_params('both', labelsize=15)
plt.show()

plt.hist((fuv_band - nuv_band)[idx_nospurious], bins=np.arange((g_band - r_band)[idx_nospurious].min(), (fuv_band - nuv_band)[idx_nospurious].max(), 0.5), color='#e6550d',
         log=True)
plt.xlabel("FUV-NUV", fontsize=15)
plt.ylabel("Frequency", fontsize=15)
plt.tick_params('both', labelsize=15)
plt.show()


I am working on the file "Coding/Catalog Analysis.ipynb".
The first task is:
Plot 07: heatmap
Can you write Python code for it?

my_dataframe = pd.read_csv('myGAMA_ALL_AB_ABSOL_MAGS.csv')
my_dataframe = pd.DataFrame(my_dataframe)
my_dataframe = my_dataframe.drop(['CATAID'], axis=1)

my_dataframe = my_dataframe.fillna(-999)

my_data_10 = my_data[:, 0:10].astype(str)
print my_data_10.shape

my_data_10 = pd.DataFrame(my_data_10).fillna(-999)
my_data_10 = my_data_10.dropna

lambda_displaced_fuv = 1530 * (1.+0.4)
print lambda_displaced_fuv


I am working on the file "Coding/Catalog Analysis.ipynb".
The first task is:
The following constraint selects UV upturn objects: (nuv_band - r_band) > 5.4) x ((fuv_band - nuv_band) < 0.9) x ((fuv_band - r_band) < 6.6)
The following constraint removes spurious data: (r_band>0) x (nuv_band>0) x (fuv_band>0)
Can you write Python code for it?

index_uvup = np.where(((nuv_band - r_band) > 5.4) * ((fuv_band - nuv_band) < 0.9) * ((fuv_band - r_band) < 6.6)*(r_band>0)*(nuv_band>0)*(fuv_band>0))
print fuv_band[index_uvup].size

plt.plot((nuv_band - r_band), (fuv_band - nuv_band), 'o', color = '#fdae6b', alpha=0.7)
plt.plot((nuv_band - r_band)[index_uvup], (fuv_band - nuv_band)[index_uvup], 'o', color = '#e6550d', alpha=0.7)
plt.axvline(x=5.4, color='black', linewidth=2.)
plt.axhline(y=0.9, color='black', linewidth=2.)
plt.xlabel("NUV-r")
plt.ylabel("FUV-NUV")
plt.xlim(-5, 10)
plt.ylim(-10, 15)
plt.savefig('./Figs/uvupturn.png', dpi=100)
plt.show()

plt.plot((nuv_band - r_band), (fuv_band - nuv_band), 'o', color = '#fdae6b', alpha=0.7)
plt.plot((nuv_band - r_band)[index_uvup], (fuv_band - nuv_band)[index_uvup], 'o', color = '#e6550d', alpha=0.7)
plt.axvline(x=5.4, color='black', linewidth=2.)
plt.axhline(y=0.9, color='black', linewidth=2.)
plt.xlabel("NUV-r")
plt.ylabel("FUV-NUV")
# plt.xlim(-5, 10)
# plt.ylim(-10, 15)
plt.savefig('./Figs/uvupturn_spurious.png', dpi=100)
plt.show()

idx_nospurious = np.where((r_band>0)*(nuv_band>0)*(fuv_band>0)*(fuv_band-nuv_band<50)*((fuv_band-nuv_band)>(-20)))
print r_band[idx_nospurious].size

plt.plot((nuv_band - r_band)[idx_nospurious], (fuv_band - nuv_band)[idx_nospurious], 'o', color = '#fdae6b', alpha=0.7)
plt.plot((nuv_band - r_band)[index_uvup], (fuv_band - nuv_band)[index_uvup], 'o', color = '#e6550d', alpha=0.7)
plt.axvline(x=5.4, color='black', linewidth=2.)
plt.axhline(y=0.9, color='black', linewidth=2.)
plt.xlabel("NUV-r")
plt.ylabel("FUV-NUV")
# plt.xlim(-5, 10)
# plt.ylim(-10, 15)
plt.savefig('./Figs/uvupturn_nospurious.png', dpi=100)
plt.show()


I am working on the file "Coding_Asg_01/Assignment_1.ipynb".
The first task is:
As seen from above figures, real world networks show power law nature for degree distribution, while random graphs show more like a poisson distribution of degree over nodes. Similar results can be seen for adjacency network of common adjectives and nouns.
Can you write Python code for it?

nodes = len(G_adjnoun_adj)
edges = np.sum(G_adjnoun_adj)/2

n = 100

random_degs = np.empty([n, nodes])

for i in range(0, n):
    random_degs[i] = get_degree(create_random_graph(nodes, edges))

random_avg_degs = np.mean(random_degs, axis=0)

plt.clf()
plt.subplot(121)
plt.hist(np.asarray(G_adjnoun_deg[0])[0], bins='auto')
plt.title("Degree Distribution for \n adjacency network of common \n adjectives and nouns")
plt.xlabel("Degree")
plt.ylabel("Frequency")
plt.subplot(122)
plt.hist(random_avg_degs, bins='auto')
plt.title("Average Degree Distribution \n for Random Controls")
plt.xlabel("Degree")
plt.ylabel("Frequency")
plt.show()


I am working on the file "Coding_Asg_01/Assignment_1.ipynb".
The first task is:
C. Elegans Neural Network
Can you write Python code for it?


I am working on the file "Coding_Asg_01/Assignment_1.ipynb".
The first task is:
Qeustion 1
Can you write Python code for it?

## Undirected Unweighted Graph
G1 = nx.Graph()
G1.add_edges_from([(1, 2), (1, 3), (3, 4), (5, 6), (7, 8), (2, 5), (8, 3), (5, 8)])
plt.clf()

pos=nx.spring_layout(G1) # positions for all nodes
nx.draw(G1, pos)
nx.draw_networkx_labels(G1, pos)
plt.title('(a) Undirected Unwieghted Graph')
plt.show()

## Undirected Weighted Graph
G2 = nx.MultiGraph()
G2.add_weighted_edges_from([(1, 2, 0.1), (1, 3, 0.7), (3, 4, 1.0), (5, 6, 0.4),
                            (7, 8, 0.3), (2, 5, 0.3), (8, 3, 1.0), (5, 8, 0.5)])
pos=nx.spring_layout(G2) # positions for all nodes
nx.draw(G2, pos)
edge_labels=dict([((u,v,),d['weight'])
             for u,v,d in G2.edges(data=True)])
nx.draw_networkx_edge_labels(G2,pos,edge_labels=edge_labels)
nx.draw_networkx_labels(G2, pos)
plt.title('(b) Undirected Wieghted Graph')
plt.show()

## Directed Unweighted Graph
G3 = nx.DiGraph()
G3.add_edges_from([(1, 2), (1, 3), (3, 4), (5, 6), (7, 8), (2, 5), (8, 3), (5, 8)])
pos=nx.spring_layout(G3) # positions for all nodes
nx.draw(G3, pos)
nx.draw_networkx_labels(G3, pos)
plt.title('(c) Directed Unweighted Graph')
plt.show()

## Directed Weighted Graph
G4 = nx.MultiDiGraph()
G4.add_weighted_edges_from([(1, 2, 0.1), (1, 3, 0.7), (3, 4, 1.0), (5, 6, 0.4),
                            (7, 8, 0.3), (2, 5, 0.3), (8, 3, 1.0), (5, 8, 0.5)])
pos=nx.spring_layout(G4) # positions for all nodes
nx.draw(G4, pos)
edge_labels=dict([((u,v,),d['weight'])
             for u,v,d in G4.edges(data=True)])
nx.draw_networkx_edge_labels(G4,pos,edge_labels=edge_labels)
nx.draw_networkx_labels(G4, pos)
plt.title('(d) Directed Wieghted Graph')
plt.show()

## Storing these graphs as adjacency matrix and adjacency list
G = {'G1 (UD-UW)': G1, 'G2 (UD-W)': G2,'G3 (D-UW)': G3,'G4 (D-W)': G4}

adjacency_matrix = dict()
edgelist = dict()
for key in G:
    adjacency_matrix[key] = nx.to_numpy_matrix(G[key])
    edgelist[key] = nx.to_edgelist(G[key])

# saving
with open('ques1_stired_networks.pickle', 'w') as f:
    pickle.dump([adjacency_matrix, edgelist], f)


I am working on the file "Coding_Asg_01/Assignment_1.ipynb".
The first task is:
Qeustion 2
Can you write Python code for it?

## Fuction to find degree from adjacency matrix

# function definition
def get_degree(adj):
    """
        This function computes the degree of each node for a given adjacency matrix
        For Uniderected Graphs:
            It prints node wise degree and returns it.
        For Directed Graphs:
            It prnts both in degree and out degree and reurns both of them.
    """
    if np.allclose(adj, adj.T, atol=1e-8):
        # print "Undirected Graph"
        deg = np.sum(adj != 0, axis=0)
    else:
        # print "Directed Graph"
        deg = list()
        deg.append(np.sum(adj != 0, axis=0).astype(np.int))
        deg.append(np.sum(adj != 0, axis=1).astype(np.int).T)
    return deg

# loading adjacency matrix which were saved in question 1
f = open('ques1_stired_networks.pickle', 'r')
adj_matrix, edge_list = pickle.load(f)

for key in adj_matrix:
    print key
    deg = get_degree(adj_matrix[key])
    if len(deg) == 2:
        print "Indegree: " + str(deg[0])
        print "Outdegree: " + str(deg[1])
    else:
        print "Degree: " + str(deg)


I am working on the file "Coding_Asg_01/Assignment_1.ipynb".
The first task is:
Question 3
Can you write Python code for it?

# Computing A square for Undirected nnweighted graph G1
A = adjacency_matrix['G1 (UD-UW)']
A = A.astype(np.int)
A_square = A*A

pos=nx.spring_layout(G1) # positions for all nodes
nx.draw(G1, pos)
nx.draw_networkx_labels(G1, pos)
plt.title('Undirected Unwieghted Graph')
plt.show()
print "A_square : \n " + np.str(A_square)


I am working on the file "Coding_Asg_01/Assignment_1.ipynb".
The first task is:
Question 6
Can you write Python code for it?

## Comparing C elegans neural network with random controls
# creating underlyting graph for directed graph
G_celegans_underlying_adj = G_celegans_adj + G_celegans_adj.T
G_celegans_underlying_adj[np.matrix.nonzero(G_celegans_underlying_adj)] = 1
G_celegans_underlying_deg = get_degree(G_celegans_underlying_adj)

nodes = len(G_celegans_underlying_adj)
edges = np.sum(G_celegans_underlying_adj)/2

n = 100

random_degs = np.empty([n, nodes])

for i in range(0, n):
    random_degs[i] = get_degree(create_random_graph(nodes, edges))

random_avg_degs = np.mean(random_degs, axis=0)

plt.clf()
plt.subplot(121)
plt.hist(np.asarray(G_celegans_underlying_deg[0])[0], bins='auto')
plt.title("Degree Distribution for \nC. Elegans Neural Network \n underlying graph")
plt.xlabel("Degree")
plt.ylabel("Frequency")
plt.subplot(122)
plt.hist(random_avg_degs, bins='auto')
plt.title("Average Degree Distribution \n for Random Controls")
plt.xlabel("Degree")
plt.ylabel("Frequency")
plt.show()


I am working on the file "Coding_Asg_01/Assignment_1.ipynb".
The first task is:
Question 7
Can you write Python code for it?

## No codes in this question. Cytoscape was used to read and display thee two real world netowrks.


I am working on the file "Coding_Asg_01/Assignment_1.ipynb".
The first task is:
Question 8
Can you write Python code for it?

## Clustering Coefficient and Average custering coefficient

def get_clust_coeff(adj, n):
    """
        This Function computes clustering coefficient of the node n
        in the graph defined by adjacency matrix adj
    """
    n_deg = np.sum(adj, axis=0)[0, n]
    if (n_deg < 2):
        return 0
    neighbours = np.nonzero(adj[:, n])[0]
    neighbour_pairs = itertools.combinations(neighbours, 2)
    triangles = 0
    for i in neighbour_pairs:
        if adj[i[0], i[1]] == 1:
            triangles += 1
    clust_coeff = (float) (triangles / (n_deg*(n_deg-1)/2))
    return clust_coeff

def get_avg_clust_coeff(adj):
    """
        This function computes average clustering coefficient of a graph respresented
        from its adjacency matrix adj
    """
    nodes = len(adj)
    clust_coeffs = np.zeros([1, nodes])[0]
    for i in range(0, nodes):
        clust_coeffs[i] = get_clust_coeff(adj, i)
    avg_clust_coeff = np.mean(clust_coeffs)
    return avg_clust_coeff

# Using adjacent nouns and words undirected graph for validation
adj = G_adjnoun_adj
print "Nodewise Clustering Coefficients (for first 10 nodes: "
for i in range(0, 10):
    print "\tNode "+ str(i) + ": \t" + str(get_clust_coeff(adj, i))
print "Average Clustering Coefficient (Using function): " + str(get_avg_clust_coeff(G_adjnoun_adj))
print "Average Clustering Coefficient (Using NetworkX): " + str(nx.average_clustering(G_adjnoun))


I am working on the file "Coding_Asg_01/Assignment_1.ipynb".
The first task is:
Question 9
Can you write Python code for it?

## Finding All Pair Shortest Path using Floyd Warshall Algorithm for the aadjacent nouns and adjectives network

def floyd_warshall(adj):
    """
        This function computes all pair shortest path for the adjacency matrix adj
    """
    shortest_paths = adj
    shortest_paths[shortest_paths == 0] = np.inf
    shortest_paths[np.diag_indices_from(shortest_paths)] = 0

    nodes = len(adj)
    for k in range(nodes):
        for i in range(nodes):
            for j in range(nodes):
                shortest_paths[i, j] = min(shortest_paths[i, j] , shortest_paths[i, k]+ shortest_paths[k, j])

    return shortest_paths

adj = G_adjnoun_adj
all_pair_shortest_paths = floyd_warshall(adj)
print "All Pair Shortest path Matrix: \n" + str(all_pair_shortest_paths)
print "\nCharacteristic path length (Self-Computed): \t" + str(np.sum(all_pair_shortest_paths) / (nodes*nodes - nodes))
print "Characteristic path length (Using NetworkX): \t" + str(nx.average_shortest_path_length(G_adjnoun))


I am working on the file "Coding_Asg_02/Assignment_2.ipynb".
The first task is:
In Yeast Protein Protein Interactome visualization generated from cytoscape, many proteins were seen to interact with a large number of other proteins. Such proteins are vital in the organism's survival and are therefore considered important. A higher degree means that that protein is more important.
Can you write Python code for it?

# Random deletions in YPI

fh=open("Y2H_union.txt", 'rb')
G_yeast = nx.read_edgelist(fh)

X = np.linspace(0, 0.8, 101)
S_avg = np.zeros_like(X)
S = np.zeros_like(X)

nodes_to_del = int(0.01*G_yeast.number_of_nodes())

for x in X:
    i = np.where(X == x)[0][0]
    nodelist = G_yeast.nodes()
#     print i
#     G_copy = G
    if x != 0:
        l = np.random.randint(G_yeast.number_of_nodes(), size=nodes_to_del)
        nodelist = G_yeast.nodes()
        for f in range(len(l)):

            try:
                G_yeast.remove_node(nodelist[l[f]])
            except nx.NetworkXError:
                pass

#         print x
#     print G_yeast.number_of_nodes()

    ass = nx.connected_components(G_yeast)

    sss = sorted(ass, key = len, reverse=True)
    S[i] = len(sss[0])
    sizes = np.zeros(len(sss)-1)
    for k in range(len(sizes)):
        sizes[k] = len(sss[k+1])
    S_avg[i] = np.mean(sizes)
#     print G.number_of_nodes()

plt.plot(X, S/S[0])
plt.title('Size of the largest cluster (Failulres)')
plt.show()
plt.plot(X, S_avg)
plt.title('Average size of fragmented components (Failure)')
plt.show()

# Targeted deletions in YPI

fh=open("Y2H_union.txt", 'rb')
G_yeast = nx.read_edgelist(fh)

X = np.linspace(0, 0.8, 101)
S_avg = np.zeros_like(X)
S = np.zeros_like(X)

nodes_to_del = int(0.01*G_yeast.number_of_nodes())

for x in X:
    i = np.where(X == x)[0][0]
    nodelist = G_yeast.nodes()
#     print i
#     G_copy = G
    if x != 0:
        for f in range(len(l)):
            nodelist = G_yeast.nodes()
            try:
#                 max(G_yeast.degree().values())
                G_yeast.remove_node(nodelist[G_yeast.degree().values().index(max(G_yeast.degree().values()))])
#                 print "deleted"
            except nx.NetworkXError:
#                 print "caught"
                pass

#         print x
#     print G_yeast.number_of_nodes()

    ass = nx.connected_components(G_yeast)

    sss = sorted(ass, key = len, reverse=True)
    S[i] =...


I am working on the file "Coding_Asg_02/Assignment_2.ipynb".
The first task is:
Question 1 - Watts and Strogatz small world network model
Can you write Python code for it?

# Regular graph initialization
nodes = 1000
k = 10

adj = np.zeros([nodes, nodes])

for i in range(0, nodes):
    for j in range(1, k/2 + 1):
        adj[i, (i+j) % nodes] = 1
        adj[(i+j) % nodes, i] = 1

edgelist = np.zeros([nodes*k/2, 2])
iterator = 0

for i in range(0, nodes-1):
    for j in range(i+1, nodes):
        if adj[i, j] == 1:
            edgelist[iterator, 0] = i
            edgelist[iterator, 1] = j
            iterator += 1

G = nx.Graph()
G.add_edges_from(edgelist)
# print adj
# print sum(adj)
# G = nx.random_regular_graph(k, nodes)
G_copy = G

# plt.clf()
# pos=nx.spring_layout(G) # positions for all nodes
# nx.draw(G, pos)
# #nx.draw_networkx_labels(G, pos)
# plt.title('Initial Regular Graph')
# plt.show()

# plt.clf()
# pos=nx.spring_layout(G) # positions for all nodes
# nx.draw(G, pos)
# nx.draw_networkx_labels(G, pos)
# plt.title('Initial Regular Graph')
# plt.show()

prob = list()
prob.append(0.0001)
while prob[-1] < 1:
    prob.append(prob[-1]*2.303)
prob = prob[0:-1]

L = np.zeros([1, len(prob)+1])[0]
C = np.zeros([1, len(prob)+1])[0]

G = G_copy

L[0] = nx.average_shortest_path_length(G)
C[0] = nx.average_clustering(G)

count = 5

L_temp = np.zeros([1, count])[0]
C_temp = np.zeros([1, count])[0]

iterator = 1
for p in prob:
    print p
    for c in range(count):
        G = G_copy
        for n in G.nodes():
            for e in G.edges(n):
                if np.random.rand() < p:
                    ## perform rewiring
                    neighbours = np.array(G.edges(n))[:,1]
                    new_node = np.random.randint(1, high=nodes+1)
                    while new_node in neighbours:
                        new_node = np.random.randint(1, high=nodes+1)
                    G.remove_edge(e[0], e[1])
                    G.add_edge(e[0], new_node)
                    if not nx.is_connected(G):
                        G.remove_edge(e[0], new_node)
                        G.add_edge(e[0], e[1])
        L_temp[c] = nx.average_shortest_path_length(G)
        C_temp[c] =...


I am working on the file "Coding_Asg_02/Assignment_2.ipynb".
The first task is:
Question 2 - Barabasi Albert Model
Can you write Python code for it?

max_nodes = 500

G = nx.Graph([[0, 1]])

for x in range(2, max_nodes):
#     print x

    p = np.float32(np.asarray(G.degree().values()))
    p = p/sum(G.degree().values())

    t = True
    while t:
        try:
            n = np.random.rand()
            m = max(filter(lambda x: x< n, p))
            t = False
        except ValueError:
            continue

#     p = p*200
    G.add_node(x)
    try:

        G.add_edge(np.where(max(filter(lambda x: x < n, p)) == p)[0][0], x)
    except IndexError:
        pass
    except ValueError:
        pass
    try:
        G.add_edge(np.where(max(filter(lambda x: x < m, p)) == p)[0][0], x)
    except IndexError:
        pass
    except ValueError:
        pass

print "Average clustering coefficient"
print nx.average_clustering(G)
print "Characteristic Path length"
print nx.average_shortest_path_length(G)

plt.semilogx(G.degree().values())
plt.title('Degree Distribution')
plt.show()


I am working on the file "Coding_Asg_02/Assignment_2.ipynb".
The first task is:
Random deletions in exp and scale-free graphs
Can you write Python code for it?

max_nodes = 100
G = nx.Graph([[0, 1]])
for x in range(2, max_nodes):
    p = np.float32(np.asarray(G.degree().values()))
    p = p/sum(G.degree().values())
    t = True
    while t:
        try:
            n = np.random.rand()
            m = max(filter(lambda x: x< n, p))
            t = False
        except ValueError:
            continue
    G.add_node(x)
    try:
        G.add_edge(np.where(max(filter(lambda x: x < n, p)) == p)[0][0], x)
    except IndexError:
        pass
    except ValueError:
        pass
    try:
        G.add_edge(np.where(max(filter(lambda x: x < m, p)) == p)[0][0], x)
    except IndexError:
        pass
    except ValueError:
        pass

plt.hist(G.degree().values())
plt.title('Degree Sequence')
plt.show()

X = np.linspace(0, 0.8, 101)

## scale free
L_sf = np.zeros_like(X)
S_sf = np.zeros_like(X)

nodes_to_del = 0.01*G.number_of_nodes()

for x in X:
    i = np.where(X == x)[0][0]
#     print i
#     G_copy = G
    if x != 0:
        G.remove_nodes_from(np.random.randint(G.number_of_nodes(), size=nodes_to_del))
    try:
        L_sf[i] = nx.average_shortest_path_length(G)
    except nx.NetworkXError:
        L_sf[i] = L_sf[i-1]
    S_sf[i] = nx.number_of_nodes(max(nx.connected_component_subgraphs(G)))
#     print G.number_of_nodes()

plt.plot(X, S_sf/max_nodes)
plt.title('Size of the giant cluster')
plt.show()

plt.plot(X[1:8], L_sf[1:8])
plt.title('Characteristic Path Length')
plt.show()

G = nx.random_regular_graph(degree, nodes)
nx.number_of_nodes(min(nx.connected_component_subgraphs(G)))


I am working on the file "Coding_Asg_02/Assignment_2.ipynb".
The first task is:
Tweaking the probability
Can you write Python code for it?

max_nodes = 500

G = nx.Graph([[0, 1]])

for x in range(2, max_nodes):
#     print x

    p = np.float32(np.asarray(G.degree().values()))
    p = np.square(p)
    p = p/sum(G.degree().values())

    t = True
    while t:
        try:
            n = np.random.rand()
            m = max(filter(lambda x: x< n, p))
            t = False
        except ValueError:
            continue

#     p = p*200
    G.add_node(x)
    try:

        G.add_edge(np.where(max(filter(lambda x: x < n, p)) == p)[0][0], x)
    except IndexError:
        pass
    except ValueError:
        pass
    try:
        G.add_edge(np.where(max(filter(lambda x: x < m, p)) == p)[0][0], x)
    except IndexError:
        pass
    except ValueError:
        pass

print "Average clustering coefficient"
print nx.average_clustering(G)
print "Characteristic Path length"
print nx.average_shortest_path_length(G)

plt.semilogx(G.degree().values())
plt.title('Degree Distribution')
plt.show()


I am working on the file "Coding_Asg_03/Assignment_3.ipynb".
The first task is:
Analysis and comclusion
When Gene Duplication is very less than tfbs dublication, most of the genes end up having all the tfbs in common, thereby giving all nodes the same degree. When gene duplication probability is comparable with tfbs dublication, the gene_pool expands, and at the sme time, more and more connections gets added (as tfbs are also dublicating). Thus we get a very richly connected network.
Question 2
Can you write Python code for it?

def get_RIG(coordinates, labels, cut_off):
    """this function computes residue interaction graphs"""
    RIG = nx.Graph()
    label_ids = range(len(labels))
    RIG.add_nodes_from(label_ids)
    for i in label_ids:
        for j in label_ids:
            if not i == j:
                if np.linalg.norm(coordinates[i] - coordinates[j]) < 7:
                    RIG.add_edge(i, j)
    return RIG

def get_LIN(RIG, threshold):
    """this function computes long range network"""
    LIN = nx.Graph(RIG)
    for e in LIN.edges():
        if not abs(e[0] - e[1]) == 1:
            if abs(e[0] - e[1]) < threshold:
                LIN.remove_edge(e[0], e[1])
    return LIN

RIG_CUTOFF = 7
LIN_THRESHOLD = 12

parser = PDBParser()
pdb_files = ['1csp.pdb', '1hrc.pdb', '1pks.pdb', '2abd.pdb','3mef.pdb']
RIG = []
LIN = []

for pdb_file in pdb_files:
    structure = parser.get_structure('pdb_file', 'pdb files/' + pdb_file)
    coordinates = []
    labels = []
    for model in structure:
        for chain in model:
            for residue in chain:
                try:
                    coordinates.append(residue['CA'].get_coord())
                    labels.append(residue.get_resname())
                except KeyError:
                    pass
            RIG.append(get_RIG(coordinates, labels, RIG_CUTOFF))
            LIN.append(get_LIN(RIG[-1], LIN_THRESHOLD))
            break     ## working on chain id A only
        break         ## Working on model id 0 only

output = [['PBD ID', 'Nodes', 'Edges (RIG)', 'L (RIG)', 'C (RIG)', 'Edges (LIN)', 'L (LIN)', 'C (LIN)']]
for i in range(len(pdb_files)):
    append_list = [pdb_files[i], RIG[i].number_of_nodes(), RIG[i].number_of_edges()]
    append_list.append(nx.average_shortest_path_length(RIG[i]))
    append_list.append(nx.average_clustering(RIG[i]))
    append_list.append(LIN[i].number_of_edges())
    append_list.append(nx.average_shortest_path_length(LIN[i]))
    append_list.append(nx.average_clustering(LIN[i]))
   ...


I am working on the file "Coding_Asg_03/Assignment_3.ipynb".
The first task is:
Degree Distribution for cases

i. Gene Duplication ≪ TFBS Duplication

ii. TFBS Duplication ≫ Gene Duplication

iii. Gene Duplication ≈ TFBS Duplication.
Can you write Python code for it?

plt.subplot(211)
G1 = nx.read_gml('gml files/network-25-10-3-0.05-0.25-0.45-0.25.gml')
plt.hist(G1.degree().values())
plt.title("Gene Duplication < < TFBS Duplication")
plt.show()
plt.subplot(212)
G2 = nx.read_gml('gml files/network-25-10-3-0.49-0.01-0.49-0.01.gml')
plt.hist(G2.degree().values())
plt.title("Gene Duplication ~ TFBS Duplication")
plt.show()


I am working on the file "Coding_Asg_03/Assignment_3.ipynb".
The first task is:
Final Coexpression network (images exported from Cytoscape)
Can you write Python code for it?

print "Genes : 15"
print "TFBS : 5"
print "TFBS per gene : 2"
print "p_gene_dublication, p_gene_deletion, p_tfbs_dublication, p_tfbs_deletion"
print "0.25, 0.25, 0.25, 0.25"
Image("gml files/network-15-5-2-0.25-0.25-0.25-0.25.gml.png")

print "Genes : 15"
print "TFBS : 50"
print "TFBS per gene : 2"
print "p_gene_dublication, p_gene_deletion, p_tfbs_dublication, p_tfbs_deletion"
print "0.25, 0.25, 0.25, 0.25"
Image("gml files/network-15-50-2-0.25-0.25-0.25-0.25.gml.png")

print "Genes : 25"
print "TFBS : 10"
print "TFBS per gene : 3"
print "p_gene_dublication, p_gene_deletion, p_tfbs_dublication, p_tfbs_deletion"
print "0.25, 0.25, 0, 0.5"
Image("gml files/network-25-10-3-0.25-0.25-0-0.5.gml.png")


I am working on the file "Coding_Asg_03/Assignment_3.ipynb".
The first task is:
Observations
1csp and 1pks proteins show high cllustering coefficients in LIN models, suggesting that they have high rate of foldings.
A high clustering coefficint makes a protein to fold quickly.
Question 3
Can you write Python code for it?

def get_Bartoli_RIG_Model(nodes, edges):
    """this function computes bartoli's model of residue interaction graphs"""
    Bartoli_RIG_Model = nx.Graph()
    Bartoli_RIG_Model.add_nodes_from(range(nodes))
    # adding backbone chain
    Bartoli_RIG_Model.add_path(range(nodes))
    # making other links
    d = {}      # dictionary key: absolute difference, values: possible pairs
    for i in range(nodes):
        for j in range(nodes):
            if abs(i-j) in d:
                d[abs(i-j)].append((i, j))
            else:
                d[abs(i-j)] = [(i, j)]
    del(d[0])        # not required
    del(d[1])        # already handled in backbone
    p = np.asarray([len(x) for x in d.values()]).astype(np.float)
    p = p/np.sum(p)
    while not nx.number_of_edges(Bartoli_RIG_Model) > edges:
        x = random.choice(d[np.random.choice(d.keys())])
    #     np.random.choice(d[np.random.choice(d.keys(), p)])
        Bartoli_RIG_Model.add_edge(x[0]-1, x[1]-1)
        Bartoli_RIG_Model.add_edge(x[0]-1, x[1])
        Bartoli_RIG_Model.add_edge(x[0]-1, x[1]+1)
        Bartoli_RIG_Model.add_edge(x[0], x[1]-1)
        Bartoli_RIG_Model.add_edge(x[0], x[1])
        Bartoli_RIG_Model.add_edge(x[0], x[1]+1)
        Bartoli_RIG_Model.add_edge(x[0]+1, x[1]-1)
        Bartoli_RIG_Model.add_edge(x[0]+1, x[1])
        Bartoli_RIG_Model.add_edge(x[0]+1, x[1]+1)
    return Bartoli_RIG_Model

## Bartoli's model for protein contact map models.

Bartoli_RIG_Model = []

for rig in RIG:
    nodes = nx.number_of_nodes(rig)
    edges = nx.number_of_edges(rig)
    Bartoli_RIG_Model.append(get_Bartoli_RIG_Model(nodes, edges))

output = [['PBD ID', 'Nodes', 'Edges (RIG)', 'L (RIG)', 'C (RIG)', 'Edges (Bartoli)', 'L (Bartoli)', 'C (Bartoli)']]
for i in range(len(pdb_files)):
    append_list = [pdb_files[i], RIG[i].number_of_nodes(), RIG[i].number_of_edges()]
    append_list.append(nx.average_shortest_path_length(RIG[i]))
    append_list.append(nx.average_clustering(RIG[i]))
   ...


I am working on the file "Coding_Asg_03/Assignment_3.ipynb".
The first task is:
Observations
Bartoli's Models, when compared with the the original RIG models show that the characteristic path length and average clustering coefficients both are comparable. Clustering coefficients is a bit lower that true values.
Can you write Python code for it?

x = []
for e in Bartoli_RIG_Model[0].edges():
    x.append(np.linalg.norm(e))

plt.hist(x, bins = 30)
plt.xlabel('cartesian distance amond amino acids')
plt.ylabel('number of amino acid contacts made')
plt.show()


I am working on the file "Coding_Asg_03/Assignment_3.ipynb".
The first task is:
Question 1
Can you write Python code for it?

## Initialization

genes = 25
tfbs_pool = 10
tfbs_per_gene = 3

gene_pool = {}
for g in range(genes):
    gene_pool[g] = random.sample(range(tfbs_pool), tfbs_per_gene)

steps_of_evolution = 100

p_gene_dublication = 0.05
p_gene_deletion = 0.25
p_tfbs_dublication = 0.45
p_tfbs_deletion = 0.25
p_vector = [p_gene_dublication, p_gene_deletion, p_tfbs_dublication, p_tfbs_deletion]

filename = "network-" + str(genes) + "-" + str(tfbs_pool) + "-" + str(tfbs_per_gene)
filename = filename + "-" + str(p_vector[0]) + "-" + str(p_vector[1]) + "-"
filename = filename + str(p_vector[2])+ "-" + str(p_vector[3]) + ".gml"

## Evolution
for s in range(steps_of_evolution):
    r = np.random.choice(len(p_vector), p = p_vector)
    if r == 0:
        print "Evolution Step : " + str (s) + " Gene Dublication"
        gene_to_dublicate = random.sample(range(genes), 1)[0]
        gene_pool[genes] = copy.deepcopy(gene_pool[gene_to_dublicate])
        genes = genes + 1;
    elif r == 1:
        print "Evolution Step : " + str (s) + " Gene Deletion"
        gene_to_delete = random.sample(range(genes), 1)[0]
        for i in range(gene_to_delete, genes-1):
            gene_pool[i] = copy.deepcopy(gene_pool[i+1])
        gene_pool.pop(genes - 1)
        genes = genes-1
        if genes == 0:
            print "Gene Pool Empty !"
            break
    elif r == 2:
#         print "Evolution Step : " + str (s) + " TFBS Dublication"
        tfbs_probability = np.array(range(0, tfbs_pool))
        for g in gene_pool:
            for value in gene_pool[g]:
                tfbs_probability[value] = tfbs_probability[value]+1
        tfbs_probability = tfbs_probability.astype(np.float)
        tfbs_probability = tfbs_probability / np.sum(tfbs_probability)
        tfbs_to_dublicate = np.random.choice(tfbs_pool, p = tfbs_probability)
        flag = False
        while not flag:
            gene_target = np.random.choice(gene_pool.keys())
            if tfbs_to_dublicate not in gene_pool[gene_target]:
               ...


I am working on the file "Coding_Asg_03/Assignment_3.ipynb".
The first task is:
Question 4
Can you write Python code for it?

def display_graph(G):
    print "Nodes : " + str(G.number_of_nodes())
    print "Edges : " + str(G.number_of_edges())
    density = G.number_of_edges() / (G.number_of_nodes()*G.number_of_nodes()/2)
    print "Sparseness : " + str(1-density)
    try:
        print "Characteristic Path Length (L) : " + str(nx.average_shortest_path_length(G))
    except Exception as e:
        print "Characteristic Path Length (L) : " + str(e)
    print "Average Clustering Coefficient (C) : " + str(nx.average_clustering(G))

    nx.draw_networkx(G)
    plt.title('Network Layout')
    plt.show()

    plt.hist(G.degree()[0])
    plt.title('Degree distribution')
    plt.show()

n_protein = 100
n_protein_domains = 50
domain_per_protein = 3
protein_domains = range(n_protein_domains)
protein = np.asmatrix([random.sample(range(n_protein_domains), domain_per_protein) for x in range(n_protein)])

G_odd_odd = nx.Graph()
G_odd_odd.add_nodes_from(range(n_protein))
for x in range(n_protein):
    for y in range(n_protein):
        if not x == y:
            if np.any(protein[x,] % 2) and np.any(protein[y,] % 2):
                G_odd_odd.add_edge(x, y)

print "ODD-ODD PIN"
print "~~~~~~~~~~~"
print ""
display_graph(G_odd_odd)

G_even_even = nx.Graph()
G_even_even.add_nodes_from(range(n_protein))
for x in range(n_protein):
    for y in range(n_protein):
        if not x == y:
            if not np.all(protein[x,] % 2) and not np.all(protein[y,] % 2):
                G_even_even.add_edge(x, y)

print "EVEN-EVEN PIN"
print "~~~~~~~~~~~~~"
print ""
display_graph(G_even_even)

G_odd_even = nx.Graph()
G_odd_even.add_nodes_from(range(n_protein))
for x in range(n_protein):
    for y in range(n_protein):
        if not x == y:
            if (np.any(protein[x,] % 2) and not np.all(protein[y,] % 2)) or (not np.all(protein[x,] % 2) and np.any(protein[y,] % 2)):
                G_odd_even.add_edge(x, y)

print "ODD-EVEN PIN"
print "~~~~~~~~~~~~~"
print ""
display_graph(G_odd_even)

def is_prime(a):
    if a < 3:
        return False
    return...


I am working on the file "Coding_and_Decoding_Lempel-Ziv-Welch_in_Python.ipynb".
The first task is:
2. Décodage de Lempel-Ziv-Welch
Can you write Python code for it?

def decodage_LZW(code_lu: typing.List[int]) -> str:
    # Mot produit en sortie, vide au début
    mot_lu = ""
    # Dictionnaire mappant les entiers à la chaîne associée
    dictionnaire = { i: chr(i) for i in range(255) }
    # On saute la valeur 256
    prochain_code = 257
    # Mot actuel, vide au début
    mot_actuel = ""

    m = len(code_lu)
    # Si le code lu est vide, on renvoie un texte vide
    if m == 0: return ""

    # On lit le premier entier du code lu
    code_actuel = code_lu[0]
    # Il est forcément déjà dans le dictionnaire, on le décode directement
    mot_actuel = dictionnaire[code_actuel]
    # Et on l'écrit
    mot_lu += mot_actuel
    # On commence avec w = mot_actuel d'une lettre
    w = mot_actuel

    for i in range(1, m):
        # On lit le ième code suivant
        code_actuel = code_lu[i]
        # S'il est déjà dans le dictionnaire, le mot actuel correspond au code donné par le dict.
        if code_actuel in dictionnaire:
            mot_actuel = dictionnaire[code_actuel]
            print("Code lu = {}, mot correspondant = {}.".format(
                code_actuel, mot_actuel))
        # Sinon c'est nécessaire la prochaine valeur non encore ajoutée dans le dictionnaire
        elif code_actuel == prochain_code:
            # le mot actuel est le mot + la lettre du début de ce mot
            mot_actuel = w + w[0]
            print("Code lu = {}, mot correspondant = {}.".format(
                code_actuel, mot_actuel))
        else:
            raise ValueError("Mauvaise compression code_actuel = {}".format(code_actuel))
        # On écrit ce mot actuel
        mot_lu += mot_actuel
        print("Ajout de {} au dictionnaire, de valeur = '{}'".format(
            prochain_code, w + mot_actuel[0]))
        # On ajoute dans le dictionnaire w + mot_actuel[0]
        dictionnaire[prochain_code] = w + mot_actuel[0]
        prochain_code += 1
        # Et le w suivant est mot_actuel
        w = mot_actuel
    return mot_lu


I am working on the file "Coding_and_Decoding_Lempel-Ziv-Welch_in_Python.ipynb".
The first task is:
Exemples :
Can you write Python code for it?


I am working on the file "Colab - Setup Guide.ipynb".
The first task is:
Install dependencies
We can now install all the required project dependencies.
Can you write Python code for it?

!pip install -r assortment-quality-for-shopping-ads/requirements.txt


I am working on the file "Colab - Setup Guide.ipynb".
The first task is:
Project configuration
The next step is to enter all the parameters relative to your project.
project_id: Is the ID if your Google Cloud Platform project. (Please note that it can be different from your project name.)
merchantid_: Is your Google Merchant Center's account ID.
regionname_: Is the GCP region where the BigQuery dataset should be created (it will be created there if you have not created it already) .
datasetname_: The name of the dataset that will be used to store the tables needed by this solution. This name will also be used to create the dataset if it does not already exist.
language: The language used to display the Shopping category names (ISO format)
country: The country used for market analysis
expirationtime_: The number of days after which the table partitions will expire
Can you write Python code for it?

project_id = "fraperez-assortiment" #@param {type:"string"}
merchant_id = "1234567890" #@param {type:"string"}
region_name = "eu" #@param ["EU", "US"] {allow-input: true}
dataset_name = "merchant_center" #@param {type:"string"}
language = "en-US" #@param {type:"string"}
country = "US" #@param {type:"string"}
expiration_time = 7 #@param {type:"integer"}


I am working on the file "Collaborative Filtering on Spark.ipynb".
The first task is:
Non-RDD Example
Can you write Python code for it?

lines = spark.read.text("data/mllib/als/sample_movielens_ratings.txt").rdd
parts = lines.map(lambda row: row.value.split("::"))
ratingsRDD = parts.map(lambda p: Row(userId=int(p[0]), movieId=int(p[1]),
                                     rating=float(p[2]), timestamp=long(p[3])))
ratings = spark.createDataFrame(ratingsRDD)
(training, test) = ratings.randomSplit([0.8, 0.2])

# Build the recommendation model using ALS on the training data
# Note we set cold start strategy to 'drop' to ensure we don't get NaN evaluation metrics
als = ALS(maxIter=5, regParam=0.01, userCol="userId", itemCol="movieId", ratingCol="rating",
          coldStartStrategy="drop")
model = als.fit(training)

# Evaluate the model by computing the RMSE on the test data
predictions = model.transform(test)
evaluator = RegressionEvaluator(metricName="rmse", labelCol="rating",
                                predictionCol="prediction")
rmse = evaluator.evaluate(predictions)
print("Root-mean-square error = " + str(rmse))

# Generate top 10 movie recommendations for each user
userRecs = model.recommendForAllUsers(10)
# Generate top 10 user recommendations for each movie
movieRecs = model.recommendForAllItems(10)


I am working on the file "Collaborative Filtering.ipynb".
The first task is:
Lastly we add up the score for each title using groupby(). We also normalize the score by dividing it with the sum of the weights. Base on other critics' similarity and their rating, we have made a movie recommendation for Toby. The number matches the result of the book.
Can you write Python code for it?

recommendations = rating_c.groupby('title').apply(lambda s: s.sim_rating.sum() / s.similarity.sum())
recommendations.sort_index(ascending=False)


I am working on the file "Collaborative Filtering.ipynb".
The first task is:
Putting it all together:
Can you write Python code for it?

rating = pd.read_csv('data/movie_rating.csv')
rp = rating.pivot_table(index=['title'], columns=['critic'], values='rating')

rating_toby = rp['Toby']
sim_toby = rp.corrwith(rating_toby)

rating_c = rating.loc[rating_toby[rating.title].isnull().values & (rating.critic != 'Toby')]
rating_c_similarity = rating_c['critic'].map(sim_toby)
rating_c = rating_c.assign(similarity=rating_c_similarity, sim_rating=rating_c_similarity * rating_c.rating)

recommendations = rating_c.groupby('title').apply(lambda s: s.sim_rating.sum() / s.similarity.sum())
recommendations.sort_index(ascending=False)


I am working on the file "Collaborative Filtering.ipynb".
The first task is:
The file contains ratings from different critics on various titles.
Can you write Python code for it?

rating = pd.read_csv('data/movie_rating.csv')
rating.head()


I am working on the file "Collaborative Filtering.ipynb".
The first task is:
We will first create the matrix with titles of movies as rows and critics as columns. Each cell contains the rating from the corresponding user for a rating.
Can you write Python code for it?

rp = rating.pivot_table(index=['title'], columns=['critic'], values='rating')
rp


I am working on the file "Collocation.ipynb".
The first task is:
As we can see from the chart above, work is the highest frequency word in our lemmatized corpus with stopwords applied, followed by right. The word privacy does not even occur in the list. The code below calculates the frequency and percentage of times these words occur in the text. While work makes up 0.56% of the total words in the text, privacy accounts for only 0.06%.
Can you write Python code for it?

print('privacy:',fd['privacy'], 'times or','{:.2%}'.format(float(colText.count("privacy"))/(len(colText))))
print('right:',fd['right'], 'times or','{:.2%}'.format(float(colText.count("right"))/(len(colText))))
print('work:',fd['work'], 'times or','{:.2%}'.format(float(colText.count("work")/(len(colText)))))


I am working on the file "Collocation.ipynb".
The first task is:
Below are the results for the Log-likelihood test. The bigrams are sorted in order of significance, and the order of the words in the word-pairs shows their placement in the text. This means that the most significant bigram in the Log-likelihood test contained the words digital privacy, in that order. The word digital appears later on in the list with a lower score when it occurs after the word privacy. Scores above 3.8 are considered to be significant for the Log-likelihood test.
Can you write Python code for it?


I am working on the file "Collocation.ipynb".
The first task is:
Collocational analysis is a frequency-based technique that uses word counts to determine significance. One of the problems with counting word frequencies, as we have seen in other sections, is that the most frequently occuring words in English are function words, like the, of, and and. For this reason, it is neccessary to remove these words in order to obtain meaningful results. In text analysis, these high frequency words are compiled into lists called stopwords. While standard stopword lists are provided by the NLTK module, for the Hansard Corpus it was necessary to remove other kinds of words, like proper nouns (names and place names), and other words common to parliamentary proceedings (like Prime Minister, Speaker, etc.). These words, along with the standard stopwords, can be seen below.
Here, we use our read_file function to read in a text file of custom stopwords, assigning the variable customStopwords. We tokenize the list using the split function and then create a variable called hansardStopwords that incorporates the NLTK stopword list, and adding the words from customStopwords if they don't already occur in the NLTK list.
Can you write Python code for it?

stopwords = read_file('../HansardStopwords.txt')
customStopwords = stopwords.split()

#default stopwords with custom words added
hansardStopwords = nltk.corpus.stopwords.words('english')
hansardStopwords += customStopwords

print(hansardStopwords)


I am working on the file "Collocation.ipynb".
The first task is:
Distribution
Before I describe the statistical tests that we will use to determine the collocates for privacy, it is important to briefly discuss distribution. The chart below maps the distribution of the top 25 terms in the 2015 file.
This is important because some of the tests assume a normal distribution of words in the text. A normal distribution means that the majority of the words occur a majority of the time; it is represented in statistics as a bell curve. This means that 68% of the words would occur within one standard deviation of the mean (or average frequency of each word in the text), 95% within two standard deviations, and 99.7 within three standard deviations.
What this means, is that tests that assume a normal distribution will work, but have inaccurate statistics to back them up. I've chosen to describe all of the collocational tests here as a matter of instruction and description, but it's important to understand the tests and what they assume before making research claims based on their results.
The code below calls on the NLTK function FreqDist. The function calculates the frequency of all the words in the variable and charts them in order from highest to lowest. Here I've only requested the first 25, though more or less can be displayed by changing the number in the brackets. Additionally, in order to have the chart displayed inline (and not as a popup), I've called the magic function matplotlib inline. iPython magic functions are identifable by the % symbol.
Can you write Python code for it?

%matplotlib inline
fd = nltk.FreqDist(colText)
fd.plot(25)


I am working on the file "Collocation.ipynb".
The first task is:
For reference, I ran an earlier test that shows the 10 most common bigrams without the stopwords removed. Duplicating this test only requires that stopwords not be removed as the text is being tokenized and cleaned. We can see that there is a clear difference in the types of results returned with and without stopwords applied. The list of words appearing above is much more interesting in terms of discourse analysis, when functional parliamentary phrases like Prime Minister and Parliamentary Secretary have been removed.
Here is a piece of code that shows how the ngram function works. It goes word by word through the text, pairing each word with the one that came before. That's why the last word in the first word pair becomes the first in the next word pair.
We assign our colText variable to the colBigrams variable by specifiying that we want to make a list of ngrams containing 2 words. We could obtain trigrams by changing the 2 in the first line of code to a 3. Then, in the second line of code, we display the first 5 results of the colBigrams variable with :5. We could display the first 10 by changing the number in the square brackets to :10, or show the top 10 results again by removing the colon.
Can you write Python code for it?

# creates a list of bigrams (ngrams of 2), printing the first 5
colBigrams = list(nltk.ngrams(colText, 2))
colBigrams[:5]


I am working on the file "Collocation.ipynb".
The first task is:
Here we check that the lemmatizer hasn't been over-zealous by determining the frequency for privacy before and after the lemmatizing function. The frequencies are the same, meaning we've not lost anything in the lemmatization.
Can you write Python code for it?

print('privacy:', text.count('privacy'))
print('privacy:', lems.count('privacy'))


I am working on the file "Collocation.ipynb".
The first task is:
Here we will check to make sure we've the bigram function has gone through and counted the entire text. Having one less ngram is correct because of the way in which the ngrams are generated word-by-word in the test above.
Can you write Python code for it?

print("Number of words:", len(lems))
print("Number of bigrams:", len(colBigrams))


I am working on the file "Collocation.ipynb".
The first task is:
Let's display this data as a table, and remove some of the extra decimal digits. Using the tabulate module, we call the variable log, set the table heading names (displayed in red), and set the number of decimal digits to 3 (indicated by floatfmt=".3f"), with the numbers aligned on the leftmost digit.
Can you write Python code for it?

print(tabulate(log, headers = ["Collocate", "Log-Likelihood"], floatfmt=".3f", numalign="left"))


I am working on the file "Collocation.ipynb".
The first task is:
Now, we use read_file to load the contents of the file for 2015. For consistency and to avoid file duplication, we're always reading the files from the same directory. Even though it was used for other sections, the data is the same. We read the contents of the text file, then remove the case and punctuation from the text, split the words into a list of tokens, and assign the words in each file to a list with the variable name text. What's new here, compared to other sections, is the additional removal of stopwords.
Can you write Python code for it?

file = '../Counting Word Frequencies/data/2009.txt'
name = get_filename(file)

# opens, reads, and tokenizes the file
text = read_file(file)
words = text.split()
clean = [w.lower() for w in words if w.isalpha()]
# removes stopwords
text = [w for w in clean if w not in hansardStopwords]


I am working on the file "Collocation.ipynb".
The first task is:
Once we have our numbers in a list, as the variable numlist, we can use the built in statistics library for our calculations. Below we've calculated the mean, standard deviation, and the variance.
These numbers prove that the numerical data has a non-normal distribution. The mean is relatively low, compared to the highest frequency word, work, which appears a total of 7588 times.
The low mean is due to the high number of low frequency words; there are 5847 words that appear only once, totalling 30% of the unique words in the entire set. The standard deviation is higher than the mean, which predicts a high variance of numbers in the set, something that is proven by the variance calculation. A large variance shows that the numbers in the set are far apart from the mean, and each other.
Can you write Python code for it?

datamean = statistics.mean(numlist)
print("Mean:", '{:.2f}'.format(statistics.mean(numlist)))
print("Standard Deviation:", '{:.2f}'.format(statistics.pstdev(numlist,datamean)))
print("Variance:", '{:.2f}'.format(statistics.pvariance(numlist,datamean)))


I am working on the file "Collocation.ipynb".
The first task is:
Part 1.2: Focused Bigram Search
In this section we will focus our search on bigrams that contain the word privacy. First, we'll load the bigram tests from the NLTK module, then, we will create a filter that only searches for bigrams containing privacy. To search for bigrams containing other words, the word privacy in the second line of code can be changed to something else.
Can you write Python code for it?

# loads bigram code from NLTK
bigram_measures = nltk.collocations.BigramAssocMeasures()

# ngrams with 'privacy' as a member
privacy_filter = lambda *w: 'privacy' not in w


I am working on the file "Collocation.ipynb".
The first task is:
Statistics
Raw Frequency
The frequency calculations determine both the actual number of occurences of the bigram in the corpus as well as the number of times the bigram occurs relative to the text as a whole (expressed as a percentage).
Student's-T
The Student's T-Score, also called the T-Score, measures the confidence of a claim of collocation and assigns a score based on that certainty. It is computed by subtracting the expected frequency of the bigram by the observed frequency of the bigram, and then dividing the result by the standard deviation which is calculated based on the overall size of the corpus.
The benefit of using the T-Score is that it considers the evidence for collocates based on the overall amount of evidence provided by the size of the corpus. This differs from the PMI score (described below) which only considers strength based on relative frequencies. The drawbacks to the T-Score include its reliance on a normal distribution (due to the incorporation of standard deviation in the calculation), as well as its dependence on the overall size of the corpus. T-scores can't be compared across corpora of different sizes.
Pointwise Mutual Information
The Pointwise Mutual Information Score (known as PMI or MI) measures the strength of a collocation and assigns it a score. It is a probability-based calculation that compares the number of actual bigrams to the expected number of bigrams based on the relative frequency counts of the words. The test compares the expected figure to the observed figure, converting the difference to a number indicating the strength of the collocation.
The benefit of using PMI is that the value of the score is not dependent on the overall size of the corpus, meaning that PMI scores can be compared across corpora of different sizes, unlike the T-score (described above).
The drawback to the PMI is that it tends to give high scores to low frequency words when they occur most often in the proximity another word.
Chi-square
The Chi-square (or x2)...
Can you write Python code for it?

# filter results based on statistical test

# calulates the raw frequency as an actual number and percentage of total words
act = finder.ngram_fd.items()
raw = finder.score_ngrams(bigram_measures.raw_freq)
# student's - t score
tm = finder.score_ngrams(bigram_measures.student_t)
# pointwise mutual information score
pm = finder.score_ngrams(bigram_measures.pmi)
# chi-square score
ch = finder.score_ngrams(bigram_measures.chi_sq)
# log-likelihood ratio
log = finder.score_ngrams(bigram_measures.likelihood_ratio)


I am working on the file "Collocation.ipynb".
The first task is:
To calculate the mean, and standard deviation, we must count the frequency of all the words in the text and append those values to a list. Since the numbers in the list will actually be represented as text (not as integers), we must add an extra line of code to map those values so they can be used mathematically, calling on the map function.
Can you write Python code for it?

fdnums = []
for sample in fd:
    fdnums.append(fd[sample])
numlist = list(map(int, fdnums))

print("Total of unique words:", len(numlist))
print("Total of words that appear only once:", len(fd.hapaxes()))
print("Percentage of words that appear only once:",'{:.2%}'.format(len(fd.hapaxes())/len(numlist)))


I am working on the file "Collocation.ipynb".
The first task is:
While the table above is nice, it isn't formated exactly the way it could be, especially since we already know that privacy is one half of the bigram. I want to format the list so I can do some further processing in some spreadsheet software, including combining the scores of the bigrams (like digital privacy and privacy digital) so I can have one score for each word.
The code below sorts the lists generated by each test by the first word in the bigram, appending them to a dictionary called prefix_keys, where each word is a key and the score is the value. Then, we sort the keys by the value with the highest score, and assign the new list to a new variable with the word privacy removed. This code must be repeated for each test.
For the purposes of this analysis, we will only output the two frequency tests and the Log-likelihood test.
Can you write Python code for it?

##################################################################
################ sorts list of ACTUAL frequencies ################
##################################################################

# group bigrams by first and second word in bigram
prefix_keys = collections.defaultdict(list)
for key, a in act:
    # first word
    prefix_keys[key[0]].append((key[1], a))
     # second word
    prefix_keys[key[1]].append((key[0], a))

# sort keyed bigrams by strongest association.
for key in prefix_keys:
    prefix_keys[key].sort(key = lambda x: -x[1])

# remove the word privacy and display the first 50 results
actkeys = prefix_keys['privacy'][:50]

##################################################################
#### sorts list of RAW (expressed as percentage) frequencies #####
##################################################################

# group bigrams by first and second word in bigram
prefix_keys = collections.defaultdict(list)
for key, r in raw:
    # first word
    prefix_keys[key[0]].append((key[1], r))
    # second word
    prefix_keys[key[1]].append((key[0], r))

# sort keyed bigrams by strongest association.
for key in prefix_keys:
    prefix_keys[key].sort(key = lambda x: -x[1])

rawkeys = prefix_keys['privacy'][:50]

##################################################################
############### sorts list of log-likelihood scores ##############
##################################################################

# group bigrams by first and second word in bigram
prefix_keys = collections.defaultdict(list)
for key, l in log:
    # first word
    prefix_keys[key[0]].append((key[1], l))
    # second word
    prefix_keys[key[1]].append((key[0], l))

# sort bigrams by strongest association
for key in prefix_keys:
    prefix_keys[key].sort(key = lambda x:...


I am working on the file "Combien_de_numeros_de_mobiles_francais_sont_des_nombres_premiers.ipynb".
The first task is:
Et donc, on peut calculer la part de nombres premiers parmi les numéros de téléphones mobiles français.
De souvenir, c'était environ 5.1%, vérifions :
Can you write Python code for it?

total_number = len(all_numbers)

print(f"Pour des numéros de téléphones, nombres entre {first_number} et {last_number} (inclus), il y a {count/total_number:%} nombres premiers.")


I am working on the file "Combien_de_numeros_de_mobiles_francais_sont_des_nombres_premiers.ipynb".
The first task is:
Réponse
Can you write Python code for it?

first_number = 6_00_00_00_00
last_number = 7_99_99_99_99

# test rapide
#last_number = first_number + 20

all_numbers = range(first_number, last_number + 1)

def count_prime_numbers_in_range(some_range):
    count = 0
    for number in some_range:
        if isprime(number):
            count += 1
    return count


I am working on the file "Combining rows w groupby, transform, or multiIndex.ipynb".
The first task is:
shift data to correct row using a multi-Index
Can you write Python code for it?

ids = list(df['id'])#make a list of ids to iterate over, before the hierarchical index

#df.type = df.type.map({'\*plot summary':'transect','\*remonitoring notes':'transect'})

df.loc[df.type =='map',['mapPhoto']]=df['url'] #moving cell values to correct column

df.set_index(['id','type'],inplace=True) # hierarchical index so we can call locations

#a hierarchical index uses a tuple. You can set values using loc.
#this format: df.loc[destination] = df.loc[source].values[0]
for item in ids:
    df.loc[(item,'*plot summary'),'mapPhoto'] = df.loc[(item,'map'),'mapPhoto'].values[0]

#generates a pink warning about performance, but oh well.

#here we are using an expression in parens to test for a condition
(df['type'].str.contains('\s') & df['note'].notnull()).value_counts()

df.url = df.url.str.replace(' ','_');df.url

df.url.head()

df['newurl'] = df.url.str.replace

df.newurl.head()

#for combining rows try something like this:
print(df.groupby('somecolumn')['temp variable'].apply(' '.join).reset_index())


I am working on the file "Combining rows w groupby, transform, or multiIndex.ipynb".
The first task is:
use groupby and transform to fill the row
Can you write Python code for it?

#since we're using string methods, NaNs won't work
mycols =['general_observations','mapPhoto','linephoto1','linephoto2','miscPhoto','site_description']
for item in mycols:
    df[item] = df[item].fillna('')

df.mapPhoto = df.groupby('id')['mapPhoto'].transform(lambda x: "%s" % ''.join(x))

df.linephoto1 = df.groupby(['id','date'])['linephoto1'].transform(lambda x: "%s" % ''.join(x))
df.linephoto2 = df.groupby(['id','date'])['linephoto2'].transform(lambda x: "%s" % ''.join(x))
df.miscPhoto = df.groupby(['id','date'])['miscPhoto'].transform(lambda x: "%s" % ''.join(x))

df['site_description'] = df['site_description'].str.strip()

df.to_csv('test.csv')
#done to here. Next, figure out what to do with linephotos, unclassified photos, and their notes.
#make column for photocaptions. When adding linephoto1, add 'note' and 'type' fields to caption column. E.g. 'linephoto1: 100line- | view east along transect.' Then join the rows in the groupby transform and add to site_description field.

df.shape

df[(df.type.str.contains('line\+'))&(df.linephoto2.str.len()<50)]

maps.str.len().sort_values()


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
Among the characters, we have Batman as the one with the biggest number of comics, followed by Spider-Man and Superman. After that, we have some other famous characters, like Captain America, Iron Man, Wolverine, Flash. Here, nothing surprising.
Can you write Python code for it?


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
Among the teams with most comics, nothing surprising either. X-Men in first, Avenger in second and Justice League in third. Then, the other teams, like Fantastic Four, Suicide Squad:
Can you write Python code for it?


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
And below, the charts for the teams:
Can you write Python code for it?

# Creation of charts related to teams
plt.figure(4,figsize=(10, 6))

plt.subplot(121) # Teams by quantity of comics
y_axis = top_teams_by_quantity['Quantity_of_comics']
x_axis = np.arange(len(y_axis))
plt.bar(x_axis, y_axis)
plt.xticks(x_axis+0.4, tuple(top_teams_by_quantity['Team_Name']), rotation=90)
plt.title('Teams by quantity of comics')
plt.tight_layout()

plt.subplot(122) # Teams by average ratings
y_axis = top_teams_by_rating['Average_Rating']
x_axis = np.arange(len(y_axis))
plt.bar(x_axis, y_axis)
plt.xticks(x_axis+0.4, tuple(top_teams_by_rating['Team_Name']), rotation=90)
plt.title('Teams by average ratings')
plt.tight_layout()

plt.show()


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
And the chart with the most rated comics:
Can you write Python code for it?

# Create chart with the previously sorted comics
plt.figure(figsize=(10, 6))
y_axis = comics_by_ratings_quantity.head(30)['Ratings_Quantity']
x_axis = np.arange(len(y_axis))

plt.bar(x_axis, y_axis)
plt.xticks(x_axis+0.5, tuple(comics_by_ratings_quantity.head(30)['Name']),
           rotation=90)
plt.show()


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
As we can see, the height of the bars is quite similar. It seems that the age classification does not make a significant effect on the ratings of a comic. If we see it with a purely mathematical view, comics with an age classification for 9+ years or for all ages get the best ratings, by a small margin. But it is not possible to view a strong relation, since it does not varies in the same way as the age classification increases or decreases.
Our next step is to see how the release of comics evolved (considering print versions) over the years. Remember that we already created a column with the year of release of the print version of the comic. The next step is basically to count the occurrences of each year in this column. Let's make a list with the years and then count the releases per year:
Can you write Python code for it?

# Create pivot table with print releases per year
print_releases_per_year = pd.pivot_table(comixology_df,
                                         values=['Name'],
                                         index=['Print_Release_Year'],
                                         aggfunc=[np.count_nonzero])
print_years = []
for index, row in print_releases_per_year.iterrows():
    print_year = int(index)
    print_years.append(print_year)
print_releases_per_year.index = print_years
print(print_releases_per_year)


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
In the next analysis, we will use only comics with more than 5 ratings. For that, we will filter the DataFrame. Then, we'll create a Pandas pivot table, so that we can visualize the quantity of comics with ratings and the average rating of this publisher. Then, we will consider as representative publishers those that have at least 20 comics with ratings. To do that, we will filter the pivot table. And finally, we will sort this table by average rating, going from the highest to the lowest. This means that the publishers on the top of the table will be the ones that have the best average rating from its comics.
Can you write Python code for it?

# Filter the original DataFrame for comics with more than 5 ratings
comics_more_than_5_ratings = comixology_df[comixology_df.Ratings_Quantity > 5]

# Create pivot table with average rating by publisher
publishers_avg_rating = pd.pivot_table(comics_more_than_5_ratings,
                                       values=['Rating'],
                                       index=['Publisher'],
                                       aggfunc=[np.mean, np.count_nonzero])

# Filter for any Publisher that has more than 20 comics rated
main_pub_avg_rating = publishers_avg_rating[publishers_avg_rating.
                                            count_nonzero.Rating > 20]
main_pub_avg_rating = main_pub_avg_rating.sort_values(by=('mean','Rating'),
                                                      ascending=False)
print(main_pub_avg_rating)


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
Just as curiosity, let's check the number of ratings in comics of each publisher, through another pivot table:
Can you write Python code for it?

# Create Pivot Table with quantity of ratings of each Publisher
marvel_dc_pivot_sums = pd.pivot_table(marvel_dc_comics,
                               values=['Ratings_Quantity'],
                               index=['Publisher'],
                               aggfunc=[np.sum])
print(marvel_dc_pivot_sums)


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
Let's consider only teams and characters that have more than 20 comics where their names are present on the title of the comic. So, let's make a filter:
Can you write Python code for it?

# Filter characters and teams DataFrame for rows where there are more than 20
# comics where the character / team name is present on the title of the comics
characters_df = characters_df[characters_df['Quantity_of_comics'] > 20]
teams_df = teams_df[teams_df['Quantity_of_comics'] > 20]


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
Next, we need to pass each name of character or team. First, let's define a DataFrame, and we'll filter so that the only rows that remain are the ones where the comic name includes the name of this character or team. Then, we'll extract some information from there. The quantity of comics will be the number of rows of the resulting DataFrame. Then, we will get the average price, rating and page count. All this information will be saved in a dictionary, and this dictionary will be appended to a character list, if it is a character, or a team list, if it is a team. In the end, we will have a list of dictionaries for characters and one for teams, and we will use them to create DataFrames:
Can you write Python code for it?

# Create empty list and dict to hold character info
character_row = {}
characters_dicts = []

for character in main_dc_characters:
    character_df = comixology_df[(comixology_df['Name'].str.contains(character)) &
                                 (comixology_df['Publisher'] == 'DC Comics')]
    character_row['Character_Name'] = character
    character_row['Quantity_of_comics'] = len(character_df)
    character_row['Average_Rating'] = np.nanmean(character_df['Rating'])
    character_row['Average_Price'] = np.nanmean(character_df['Original_price'])
    character_row['Average_Pages'] = np.nanmean(character_df['Page Count'])
    character_row['Publisher'] = "DC Comics"
    characters_dicts.append(character_row)
    character_row = {}

for character in main_marvel_characters:
    character_df = comixology_df[(comixology_df['Name'].str.contains(character)) &
                                 (comixology_df['Publisher'] == 'Marvel')]
    character_row['Character_Name'] = character
    character_row['Quantity_of_comics'] = len(character_df)
    character_row['Average_Rating'] = np.nanmean(character_df['Rating'])
    character_row['Average_Price'] = np.nanmean(character_df['Original_price'])
    character_row['Average_Pages'] = np.nanmean(character_df['Page Count'])
    character_row['Publisher'] = "Marvel"
    characters_dicts.append(character_row)
    character_row = {}

characters_df = pd.DataFrame(characters_dicts)

# Create empty list and dict to hold team info
team_row = {}
teams_dicts = []

for team in dc_teams:
    team_df = comixology_df[(comixology_df['Name'].str.contains(team)) &
                                 (comixology_df['Publisher'] == 'DC Comics')]
    team_row['Team_Name'] = team
    team_row['Quantity_of_comics'] = len(team_df)
    team_row['Average_Rating'] = np.nanmean(team_df['Rating'])
    team_row['Average_Price'] = np.nanmean(team_df['Original_price'])
    team_row['Average_Pages'] = np.nanmean(team_df['Page Count'])
    team_row['Publisher'] = "DC Comics"
   ...


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
Note that the giants, Marvel and DC Comics, are not among the ones in the top. If we see the complete table, they are between the middle and the bottom of the table.
To help in the visualization, let's create a matplotlib chart that represents the table above:
Can you write Python code for it?

# Create chart with average ratings for the Publishers
plt.figure(figsize=(10, 6))
y_axis = main_pub_avg_rating['mean']['Rating']
x_axis = range(len(y_axis))

plt.bar(x_axis, y_axis)
plt.xticks(x_axis, tuple(main_pub_avg_rating.index),rotation=90)
plt.show()


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
Now, let's use the iterrows() function of the DataFrame to extract the publishing year of the print version of the comic. This function creates a for loop that iterates over each row of the DataFrame. Let's use the split() function to turn the string that contains the print release date into a list of values, and the third one will be the year. In some cases, this will return a value bigger than 2016, and since this is impossible, we will define these cases as NaN:
Can you write Python code for it?

# Extract the year of release for print version
print_dates = []
for index, row in comixology_df.iterrows():
    if type(comixology_df.ix[index]['Print Release Date']) == float:
        row_year = np.nan
    else:
        row_year = int(comixology_df.ix[index]['Print Release Date'].split()[2])
        if row_year > 2016:
            row_year = np.nan
    print_dates.append(row_year)

comixology_df['Print_Release_Year'] = pd.Series(print_dates,
                                                index=comixology_df.index)


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
On the ratings, the top 3 is formed by the All-Star Squadron, from DC Comics, Fantastic Four and the Thunderbolts, from Marvel. X-Men, Avenger and Suicide Squad are in the end of the list.
Below we plot the charts for these numbers for the characters:
Can you write Python code for it?

# Create charts related to the characters information
plt.figure(3,figsize=(10, 6))

plt.subplot(121) # Characters by quantity of comics
y_axis = top_characters_by_quantity['Quantity_of_comics']
x_axis = np.arange(len(y_axis))
plt.bar(x_axis, y_axis)
plt.xticks(x_axis+0.4, tuple(top_characters_by_quantity['Character_Name']),
                             rotation=90)
plt.title('Characters by quantity of comics')
plt.tight_layout()

plt.subplot(122) # Characters by average rating
y_axis = top_characters_by_rating['Average_Rating']
x_axis = np.arange(len(y_axis))
plt.bar(x_axis, y_axis)
plt.xticks(x_axis+0.4, tuple(top_characters_by_rating['Character_Name']),
                             rotation=90)
plt.title('Characters by average ratings')
plt.tight_layout()

plt.show()


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
One thing that I believed that could make a difference in the ratings of a comic was the age classification. Were comics made to the adults rated better? Or worse? Let's check that making another pivot table:
Can you write Python code for it?

# Create pivot table with Rating by Age Rating
rating_by_age = pd.pivot_table(comics_more_than_5_ratings,
                               values=['Rating'],
                               index=['Age Rating'],
                               aggfunc=[np.mean, np.count_nonzero])

print(rating_by_age)


I am working on the file "Comixology Analysis Notebook - English.ipynb".
The first task is:
To simplify and have a better table and chart, let's consider now only the publishers that have 300 comics with ratings. First, the table:
Can you write Python code for it?

# Filter for Publishers that have more than 300 comics rated
big_pub_avg_rating = publishers_avg_rating[publishers_avg_rating.
                                           count_nonzero.Rating > 300]
big_pub_avg_rating = big_pub_avg_rating.sort_values(by=('mean','Rating'),
                                                    ascending=False)
print(big_pub_avg_rating)

# Create chart with average ratings for Publishers with more than 300 comics
# rated
plt.figure(figsize=(10, 6))
y_axis = big_pub_avg_rating['mean']['Rating']
x_axis = np.arange(len(y_axis))

plt.bar(x_axis, y_axis)
plt.xticks(x_axis+0.5, tuple(big_pub_avg_rating.index), rotation=90)
plt.show()


I am working on the file "Comixology Analysis Notebook - Português.ipynb".
The first task is:
Agora, definiremos o número máximo de colunas de cada campo para impressão da tabela em 40 colunas. Faremos isto pois o nome de alguns comics é bastante extenso, e a impressão da tabela ficaria bem esquisita. Desta forma, pelo menos conseguimos ver algumas informações a mais.
Depois vamos listar as comics que possuem 5 estrelas, possuem mais de 20 avaliações (para captar apenas as mais representativas; comics com 5 de avaliação média mas que possuam apenas uma avaliação podem não ser uma métrica muito boa) e vamos ordena-las por preço por página. No topo teremos algumas comics que são gratuitas (as 6 primeiras). Depois, temos ótimas comics, na visão dos usuários, com um preço por página bem atrativo.
Can you write Python code for it?

# Definir o número de colunas para impressão de tabelas
pd.set_option('display.max_colwidth', 40)

# Vamos listar as comics com rating 5 estrelas que possuam pelo menos 20 ratings
# e ordena-las por preço por página
comics_with_5_stars = comixology_df[comixology_df.Rating == 5]
comics_with_5_stars = comics_with_5_stars[comics_with_5_stars.
                                          Ratings_Quantity > 20]
print(comics_with_5_stars[['Name','Publisher','Price_per_page']].
      sort_values(by='Price_per_page'))