I am working on the file "DistantSupervision.ipynb".
The first task is:
Exact SVM solution
Assuming we know what the positive points are.
Can you write Python code for it?

# Beta is the coefficients, and e is the slack.
beta = Variable(D)
ea, e_ = Variable(Na), Variable(N_)
loss = 0.5 * norm(beta, 2) ** 2 + 1*sum_entries(e_)/N_ + 1* sum_entries(ea)/Na
constr = [mul_elemwise(-1,X_*beta) > 1 - e_, mul_elemwise(1,Xa*beta) > 1 - ea]
prob = Problem(Minimize(loss), constr)
print("loss", prob.solve())

w_ = np.array(beta.value).flatten()
w_ = w_ / np.linalg.norm(w_) # Until I add a 0.
print("error", np.linalg.norm(w-w_))

# Visualize data
plt.plot(np.arange(-3,3), -w[0]/w[1] * np.arange(-3,3), linestyle='dashed', c='black')
plt.plot(np.arange(-3,3), -w_[0]/w_[1] * np.arange(-3,3), c='green')
plt.scatter(X[y<0,0,0], X[y<0,1,0], c='r', marker='o')
plt.scatter(X[y<0,0,1], X[y<0,1,1], c='r', marker='o')
plt.scatter(X[y>0,0,0], X[y>0,1,0], c='b', marker='D')
plt.scatter(X[y>0,0,1], X[y>0,1,1], c='b', marker='D')


I am working on the file "DistantSupervision.ipynb".
The first task is:
Method
Can you write Python code for it?


I am working on the file "DistantSupervision.ipynb".
The first task is:
Naive
Assumes that all positive data is the same.
Can you write Python code for it?

# Beta is the coefficients, and e is the slack.
beta = Variable(D)
ep, e_ = Variable(Np), Variable(N_)
loss = 0.5 * norm(beta, 2) ** 2 + 1*sum_entries(e_)/N_ + 1* sum_entries(ep)/Np
constr = [mul_elemwise(-1,X_*beta) > 1 - e_, mul_elemwise(1,Xp*beta) > 1 - ep]
prob = Problem(Minimize(loss), constr)
print("loss", prob.solve())

w_ = np.array(beta.value).flatten()
w_ = w_ / np.linalg.norm(w_) # Until I add a 0.
print("error", np.linalg.norm(w-w_))

# Visualize data
plt.plot(np.arange(-3,3), -w[0]/w[1] * np.arange(-3,3), linestyle='dashed', c='black')
plt.plot(np.arange(-3,3), -w_[0]/w_[1] * np.arange(-3,3), c='green')
plt.scatter(X[y<0,0,0], X[y<0,1,0], c='r', marker='o')
plt.scatter(X[y<0,0,1], X[y<0,1,1], c='r', marker='o')
plt.scatter(X[y>0,0,0], X[y>0,1,0], c='b', marker='D')
plt.scatter(X[y>0,0,1], X[y>0,1,1], c='b', marker='D')


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
A se observa ca in $0$ densitatea ia valoarea $1/\theta$. Deci pe masura ce $\theta$ creste,
$1/\theta$ descreste.
Functia de repartitie a distributiei $Exp(\theta)$:
    $F(x)=\left{\begin{array}{ll}0 & x<0\1-e^{-x/\theta}& x\geq 0\end{array}\right.$
Can you write Python code for it?

def Expocdf(theta):
    V=st.expon(scale=theta)
    x=np.linspace(0,10,100)
    y=V.cdf(x)
    fig, ax = plt.subplots()
    ax.plot(x,y)
    ax.fill_between(x, y,  alpha=.33)
    ax.set_title(r'Variatia functiilor de repartitie $Exp(\theta)$')
    ax.set_xlim(0,10)
    ax.set_ylim(0, 1.0)

interact(Expocdf, theta=(0.2, 3.2))


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Am analizat in Cursul 15 media si dispersia distributiei Pareto si am concluzionat ca o variabila aleatoare
$X\sim Pareto(\alpha,\beta)$ are medie doar pentru $\alpha>1$ si in acest caz media este:
$$M(X)=\displaystyle\frac{\alpha\beta}{\alpha-1}$$
Variabila $X$ are dispersie doar pentru $\alpha>2$ si:
    $$\sigma^2(X)=\displaystyle\frac{\alpha\beta^2}{(\alpha-1)^2(\alpha-2)}$$
Remarcam ca daca $V\sim Pareto(\alpha, 1)$ are distributia Pareto standard, atunci
$M(X)=\beta M(V)$, iar $\sigma^2(X)=\beta^2\sigma^2(V)$.
In concluzie:
- pentru  $\alpha\in(0,1]$,  o variabila aleatoare Pareto nu are medie nici dispersie.
- pentru $\alpha\in(1,2]$, o variabila aleatoare Pareto are medie finita si dispersie infinita
- pentru $\alpha>2$,  variabila are si medie si dispersie finita.
Distributia Pareto de medie finita si dispersie infinita a fost identificata ca
fiind distributia de probabilitate:
- a duratei  serviciului procesorului pentru procesele sistemului de operare UNIX
- a  dimensiunii fisierelor stocate pe serverele WEB
- a lungimii intervalului de timp intre doua pachete succesive  de informatie in  traficul internet
Sa ilustram interactiv proprietatile deduse pentru media si dispersia distributiei Pareto:
Can you write Python code for it?

plt.rcParams['figure.figsize'] = 5, 3

beta=0.5
def  ParetoMeanVar(alpha):
    V=st.pareto(alpha)# beta setat pe 1
    print 'media=', beta*V.mean()
    print 'dispersia=', beta**2*V.var()

interact(ParetoMeanVar, beta=0.5, alpha=(0.1, 3))


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Daca insa studiem o variabila aleatoare $X\sim Pareto(\alpha, \beta\neq 1)$, atunci exploatam relatiile de mai sus.
De exemplu  sa trasam densitatea de probabilitate a variabilei aleatoare $X$ de parametri $\alpha=3$ si $\beta=2$:
Can you write Python code for it?

beta=2.0
x=np.linspace(0,20, 100)
y=(1.0/beta)*V.pdf(x/beta)
plt.plot(x,y)
plt.fill_between(x, y,  alpha=.33)
plt.title('Densitatea de probabilitate Pareto')


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Distributia Pareto
Distributia Pareto a fost introdusa de economistul italian V Pareto, ca distributie a veniturilor/averii
intr-o tara, oras, etc. El a formulat principiul 80/20, conform caruia $20\%$ din populatia unei tari
detine $80\%$ din averea totala.
Acest principiu se  manifesta si in traficul internet: $20\%$ din paginile WEB, atrag $80\%$ din traficul internet.
Densitatea ei de probabilitate depinde
 doi parametri $\alpha>0$ si $\beta> 0$:
        $$f(x)=\left{\begin{array}{ll}  0& x<\beta\
                     \displaystyle\frac{\alpha}{\beta}\left(\frac{\beta}{x}\right)^{\alpha+1}&x\geq \beta\end{array}\right.$$
In scipy.stats este implementata distributia Pareto corespunzatoare lui $\beta=1$, respectiv densitatea de probabilitate, functia de repartitie,  inversa ei, media, dispersia. Pentru a implementa/apela corect aceste functii pentru o variabila aleatoare Pareto  arbitrara,
notam cu
$V$ o variabila aleatoare ce are distributia $Pareto(\alpha, \beta=1)$ si cu $X$ o variabila aleatoare ce are distributia Pareto, $Pareto(\alpha, \beta \neq 1)$.  Sa deducem relatia intre densitatile de probabilitate $f_X$ si $f_V$, respectiv functiile de repartitie
$F_X$ si $F_V$, precum si intre invesersele lor.
$F_V(x)= 1-1/x^\alpha$, $x\geq 1$, iar  $F_X(x)=1-(\beta/x)^{\alpha}$, $\forall\: x\geq \beta$.
Prin urmare:
$$F_X(x)=F_V(x/\beta), \quad\forall\: x\geq \beta$$
Derivand aceasta egalitate functionala obtinem:
relatia intre densitatile de probabilitate:
$$f_X(x)=\displaystyle\frac{1}{\beta}f_V(x/\beta), \quad \forall\: x\geq \beta$$
Inversa functiei de repartitie, ${F_V}{|[1,\infty)}$, este $F_V^{-1}(u)=(1-u)^{-1/\alpha}$, iar
a lui ${F_X}{|[\beta,\infty)}$, $F_X^{-1}(u)=\beta F_V^{-1}(u)$.
Declaram o variabila aleatoare Pareto standard (i.e. corespunzatoare lui $\beta=1$) astfel:
Can you write Python code for it?


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Fixand  $\alpha=0.8$ si variind $\beta$ avem:
Can you write Python code for it?

fig, ax = plt.subplots()
alpha=0.8
betaVals=np.arange(0.2, 2.2, 0.4)
for beta in betaVals:
    x=np.linspace(beta,20, 200)
    y=(1.0/beta)*st.pareto.pdf(x/beta,alpha)
    ax.plot(x,y, lw=2,  label=r'$\beta = %.1f $'%beta)
    ax.set_title(r'Densitati ale distributiiilor de probabilitate $Pareto(0.8, \beta)$')
    ax.legend()


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Fixand parametrul  $\sigma$ si variind media, $m$, graficul densitatii corespunzatoare se translateaza
pe directia axei Ox. De aceea media se numeste si parametru de locatie (a axei de simetrie a graficului, in acest caz).
Can you write Python code for it?

def MedieVar(m):
   sigma=1
   x=np.linspace(m-4*sigma, m+4*sigma, 100)
   y=st.norm.pdf(x,loc=m, scale=sigma)
   fig, ax = plt.subplots()
   ax.plot(x,y)
   ax.fill_between(x, y,  alpha=.33)
   ax.set_title(r'Variatia graficului densitatii $N(m,1)$')
   ax.set_xlim(-4, 6)

interact(MedieVar, m=(0,2, 0.2))


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Histograma valorilor generate are  alura graficului densitatii mixturii.
Histogramele de culoare ale imaginilor digitale au deobicei distributia unei mixturi Gaussiene.
Can you write Python code for it?

from IPython.core.display import HTML
def  css_styling():
    styles = open("./custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Mixtura Gaussiana 1D
Sa construim o mixtura din trei distributii normale, $f=p_0f_0+p_1f_1+p_2f_2$, numita mixtura Gaussiana:
Can you write Python code for it?

p=[0.3, 0.48, 0.22]#probabilitatile din definitia mixturii
m=[-4.8, -0.5, 3.5]#lista mediilor celor 3 distributii
sigma=[0.8, 1.0, 1.7]# lista abaterilor standard
x=np.linspace(-10,10, 1000)
y=np.zeros(x.shape)
for i in range(3):
    y+=p[i]*st.norm.pdf(x,loc=m[i], scale=sigma[i])
plt.plot(x,y)
plt.fill_between(x, y,  alpha=.33)

def  simDiscrete(pr):
    k=0
    F=pr[0]
    u=np.random.random()
    while(u>F):
        k+=1
        F=F+pr[k]
    return k

def  GaussianMixture1D(N, p, m,sig):
   dis=[simDiscrete(p) for i in xrange(N)]#genereaza N observatii asupra distr discrete

   vals=np.empty(N, dtype=float)# vals va contine valorile generate ale mixturii
   n=len(p)
   for k in range(n):
       I=[j for j in range(N) if dis[j] == k]# lista indicilor elementelor listei dis, egale
                                             #cu k (k=0, 1,  ..., m-1)
       s=len(I)
       va=st.norm.rvs(size=s, loc=m[k], scale=sig[k])#generam atatea valori din f_k
                                                     #cat este lungimea lui I
       vals[I]=va # in pozitiile I copiem valorile generate
   return vals


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
O alta modalitate de a vizualiza variatia graficului densitatii, cand unul din parametri este fix
si celalalt variabil, consta in a colora diferit graficele corespunzatoare la parametri diferiti.
Fixam mai intai pe $\beta=1$ si desenam grafice pentru valori crescatoare ale lui $\alpha$:
Can you write Python code for it?

plt.rcParams['figure.figsize'] = 8, 5
fig, ax = plt.subplots()
beta=1
alphaVals=np.arange(0.2, 2.2, 0.4)
for alpha in alphaVals:
    x=np.linspace(beta,20, 100)
    y=st.pareto.pdf(x,alpha)
    ax.plot(x,y, lw=2, label=r'$\alpha = %.1f $'%alpha)
    ax.set_title(r'Densitati ale distributiilor de probabilitate $Pareto(\alpha, 1)$')
    ax.legend()


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
O alta modalitate este  sa nu inghetam definitia, ci sa apelam  metodele pentru o variabila aleatoare,
st.expon, cu argumentele
 si  cuvintele cheie caracteristice, distributiei ei.
De exemplu,  pentru a trasa functia de repartitie a variabilei corespunzatoare lui theta=1.5 sau pentru a calcula
dispersia pentru $Exp(1.5)$   apelam metodele aferente unei variabile aleatoare st.expon, astfel:
Can you write Python code for it?

theta=1.5
x=np.linspace(0,20, 100)
y=st.expon.cdf(x,scale=theta)
plt.title('Functia de repartitie a distributiei Exp(1.5)')
plt.plot(x,y)

print 'Dispersia pentru Exp(1.5):', st.expon.var(scale=theta)


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
O variabila aleatoare, $X\sim N(m,\sigma)$, normal distribuita de parametri $m, \sigma$, arbitrari, are densitatea de probabilitate:
$$f_X(x;m,\sigma)=\displaystyle\frac{1}{\sigma\sqrt{2\pi}}e^{-(x-m)^2/2\sigma^2}, x\in\mathbb{R}$$
Can you write Python code for it?

m=2
sigma=0.75
W=st.norm(loc=m, scale=sigma)
xx=np.linspace(m-4*sigma, m+4*sigma, 100)
plt.title('Densitatea de probabilitate N(2, 0.75)')
plt.plot(xx,W.pdf(xx))


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Pentru a le folosi in acest notebook in locul culorilor oferite de matplotlib, le salvam in
colors:
Can you write Python code for it?

colors=seaborn.color_palette()# colors[i] va fi culoarea i, i=0,...5


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Prin urmare $z_{0.4}\approx-0.25$, iar $z_{0.78}\approx 0.77$.
Histograma valorilor generate de  simulatorul distributiei normale $N(1.5, 0.8)$:
Can you write Python code for it?

plt.rcParams['figure.figsize'] = 5, 3
X=st.norm(loc=1.5, scale=0.8)
fig,ax=plt.subplots()
ax.set_xlim(-1.5, 4.5)
vals=X.rvs(size=1000)
histo=plt.hist(vals, bins=50, normed=True)
x=np.linspace(np.min(vals), np.max(vals), 100)
ax.plot(x, X.pdf(x), color=colors[2])


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Sa calculam $\alpha$-cvantila distributiei $N(0,1)$, pentru $\alpha=0.4$ si $\alpha=0.78$:
Can you write Python code for it?

print 'alpha cvantila 0.4 si respectiv 0.78 este:', st.norm.ppf([0.4, 0.78])


I am working on the file "DistributiiProbCont.ipynb".
The first task is:
Sa ilustram acum dependenta graficului densitatii Pareto de cei doi parametri, $\beta$ si $\alpha$:
Can you write Python code for it?

def Paretopdf(beta, alpha):
    x=np.linspace(0,20,100)
    y=(1.0/beta)*st.pareto.pdf(x/beta, alpha)
    fig, ax = plt.subplots()
    ax.plot(x,y)
    ax.fill_between(x, y,  alpha=.33)
    ax.set_title(r'Variatia densitatilor de probabilitate $Pareto(\alpha, \beta)$')
    ax.set_ylim(0,3)

interact(Paretopdf, beta=(0.2, 2, 0.2), alpha=(0.1, 3, 0.3))


I am working on the file "Distribution regression.ipynb".
The first task is:
Import data
Can you write Python code for it?

def load_data():
    full_data = pd.read_csv("Data/X.csv")
    train_y = pd.read_csv("Data/y_train.csv")
    # Rename columns to something more interpretable
    columns = (["reflectance_" + str(i) for i in range(7)]
               + ["solar_" + str(i) for i in range(5)] + ["id"])
    full_data.columns = columns

    # Move ID column to the beginning
    id_column = full_data["id"]
    full_data.drop(labels=["id"], axis=1, inplace = True)
    full_data.insert(0, "id", id_column)

    # Add the target value column to the training part
    # in full_data
    split = 98000
    y_id_dict = train_y.set_index("id")["y"].to_dict()
    full_data.loc[:(split-1), "y"] = full_data.loc[:(split-1), "id"].map(y_id_dict)

    # Split into training and testing data
    train, test = full_data[:split], full_data[split:]
    return (train, test)

train, test = load_data()


I am working on the file "Distribution regression.ipynb".
The first task is:
Kernel selection
A kernel is characterised by a parameter we will call $\theta$ and the ridge regression depends on the L2 regularisation $\lambda$. Through cross-validation, we selected the kernels giving the most stable validation loss. They are given below with their associated parameters:

Cauchy:
$$k_C(a,b) = \dfrac{1}{1 + \dfrac{\Vert a-b\Vert_2^2}{\theta^2}}, \quad\theta_C = 16, \quad\lambda_C = 2^{-23} $$
Matérn 5/2:
$$k_M(a,b) = \left(1 + \dfrac{\sqrt{5}\Vert a-b\Vert_2^2}{\theta} + \dfrac{5\Vert a-b\Vert_2^2}{3\theta^2} \right)e^{-\dfrac{\sqrt{5}\Vert a-b\Vert_2^2}{\theta}}, \quad\theta_M = 64, \quad\lambda_M = 2^{-31} $$
Rational quadratic:
$$k_r(a,b) = 1 - \dfrac{\Vert a-b\Vert_2^2}{\Vert a-b\Vert_2^2 + \theta}, \quad\theta_r = 512, \quad\lambda_r = 2^{-26}$$

2. Stacking
We will then map the features in the three spaces that describes the data in different ways. Each kernel ridge regression gives a prediction of the labels and combining them might give a better result for three reasons:

Statistical reason: we might not have enough data and even if each model $h_i$ performs well on the training set, the true model $f$ might still be not reached ;
Computational reason: each model $h_i$ only finds a local optima ;
Representational reason: the true model is out of the representation of functions we're considering.

Combining our model might take us a step closer to finding the true model $f$. The ensembling technique we used was out-of-fold stacking.
Out-of-fold prediction
In the first stage, out-of-fold prediction is applied to ensure that each first-layer regressor does not overfit by predicting on data already seen. For each regressor, we iteratively separate the training data in $N$ folds ($N=5$ in our model), and then use N-1 folds to train the model and then predict the target value of the remaining fold. To create the new testing set, the average of the predictions of each fold is taken.
Can you write Python code for it?

# G_train and G_test are pandas dataframes
# krr is a kernel ridge regression
def oof_prediction(krr, G_train, y_train, G_test, n_folds, random_seed):
    kf = KFold(n_splits=n_folds, shuffle=True, random_state=random_seed)
    n_train = G_train.shape[0]
    n_test = G_test.shape[1]
    oof_train = np.zeros(n_train)
    oof_test = np.zeros(n_test)
    oof_test_folds = np.zeros((n_test, n_folds))

    for i, (train_index, test_index) in enumerate(kf.split(G_train)):
        G_tr = G_train.loc[train_index, train_index].values
        y_tr = y_train[train_index].reshape((1, -1))
        G_te = G_train.loc[train_index, test_index].values

        krr.fit(G_tr, y_tr)
        oof_train[test_index] = krr.predict(G_te)
        G_test_partial = G_test.loc[train_index, :]
        oof_test_folds[:, i] = krr.predict(G_test_partial.values)

    oof_test = oof_test_folds.mean(axis=1)
    return oof_train, oof_test

nb_bags_train = 980
# Create a vector with the unique values of y for each ID.
y_train = train.groupby("id")["y"].median().values

# Load Gram matrices
def load_gram(csv_file, nb_bags_train):
    # Import data
    G = pd.read_csv(csv_file, header=None)
    idx_train = nb_bags_train - 1
    idx_test = nb_bags_train
    G_train = G.loc[:idx_train, :idx_train]
    G_test = G.loc[:idx_train, idx_test:]
    return (G_train, G_test)

# Define models and import Gram matrices
# Cauchy
l2_reg_cauchy = 2**(-23)
cauchy = RidgeRegression(l2_reg_cauchy)
G_train_cauchy, G_test_cauchy = load_gram("kernels_me/Cauchy_16.csv", nb_bags_train)

# Matern 5/2
l2_reg_matern_52 = 2**(-31)
matern_52 = RidgeRegression(l2_reg_matern_52)
G_train_matern_52, G_test_matern_52 = load_gram("kernels_me/Matern_52_64.csv", nb_bags_train)

# Rational quadratic
l2_reg_rquadr = 2**(-26)
rquadr = RidgeRegression(l2_reg_rquadr)
G_train_rquadr, G_test_rquadr = load_gram("kernels_me/rquadr_512.csv", nb_bags_train)

# Create OOF train and test predictions
# Cauchy
cauchy_oof_train, cauchy_oof_test = oof_prediction(cauchy, G_train_cauchy,
                ...


I am working on the file "Distribution regression.ipynb".
The first task is:
Second stage prediction with SVR
In the second stage, a Support Vector Regression is fed with the different predictions from the kernel ridge regression to predict the target value $y$.
The SVR uses these predictions to compute the optimal weights assigned to each kernel regression and we might hope to find a better optimum to approximate the true regression function $f$.
Can you write Python code for it?

# Building the new data frames using the
# of out-of-fold predictions
kernel_train = pd.DataFrame({'cauchy': cauchy_oof_train,
                             'matern_52': matern_52_oof_train,
                             'rquadr': rquadr_oof_train})

kernel_train["y"] = y_train

kernel_test = pd.DataFrame({'cauchy': cauchy_oof_test,
                            'matern_52': matern_52_oof_test,
                            'rquadr': rquadr_oof_test})

cols_excl_kernel = ["y"]
cols_kernel = [c for c in kernel_train.columns if c not in cols_excl_kernel]

kernel_train.head()


I am working on the file "Distribution regression.ipynb".
The first task is:
Set parameters
Can you write Python code for it?

random_seed = 8
# Set folds for out-of-fold prediction
n_folds = 5


I am working on the file "Distribution regression.ipynb".
The first task is:
Tuning SVR's parameters
Can you write Python code for it?

# Root mean squared error metric
def RMSE(y, y_hat):
    out = np.sqrt(mean_squared_error(y.reshape((-1,)), y_hat.reshape((-1,))))
    return (out)

def scoring_function(parameters):
    print("Training the model with parameters: ")
    print(parameters)

    # Run several KFold shuffles and take the mean RMSE
    average_RMSE = []
    nb_run = 10

    for m in range(nb_run):
        KFold_RMSE = 0.0
        n_splits = 5

        kf = KFold(n_splits=n_splits, shuffle=True, random_state=(random_seed+m))
        nb_fold = 0
        for train_index, validation_index in kf.split(kernel_train):
            nb_fold += 1
            train_fold, validation_fold = kernel_train.loc[train_index], kernel_train.loc[validation_index]

            svr = SVR(C=parameters["C"], epsilon=parameters["epsilon"])
            svr.fit(train_fold[cols_kernel], train_fold["y"])

            y_hat_test = svr.predict(validation_fold[cols_kernel])
            RMSE_test = RMSE(y_hat_test, validation_fold["y"].values)

            KFold_RMSE += RMSE_test

        KFold_RMSE /= n_splits

        average_RMSE.append(KFold_RMSE)

    average_RMSE = np.array(average_RMSE)

    print("Cross-validation score: {0} +/- {1}\n".format(average_RMSE.mean(),
                                                         2*average_RMSE.std()))

    return {"loss": average_RMSE.mean(), "status": STATUS_OK}

# Grid to pick parameters from.
parameters_grid = {"C":       hp.choice("C", np.arange(0.5, 3, 0.5)),
                   "epsilon": hp.choice("epsilon", np.arange(0.05, 0.25, 0.05))
                  }
# Record the information about the cross-validation.
trials = Trials()

best = fmin(scoring_function, parameters_grid, algo=tpe.suggest, max_evals=10,
            trials=trials)

min(trials.losses())

# Save the best parameters as a csv.
best_parameters = pd.DataFrame({key: [value] for (key, value) in
                                zip(space_eval(parameters_grid, best).keys(),
                                   ...


I am working on the file "Distributions/DensityDist Example.ipynb".
The first task is:
Given example
https://pymc3.readthedocs.io/en/latest/notebooks/normalizing_flows_overview.html?highlight=densitydist
Can you write Python code for it?

from pymc3.distributions.dist_math import bound

def pot1(z):
    z = z.T
    return .5*((z.norm(2, axis=0)-2.)/.4)**2 - tt.log(tt.exp(-.5*((z[0]-2.)/.6)**2) +
                                                      tt.exp(-.5*((z[0]+2.)/.6)**2))

def cust_logp(z):
    # this works to cut things off, useful or not but 0 is a nature place a lot of times
    return bound(-pot1(z), z>-5, z<5)
#     return -pot1(z)

with pm.Model() as pot1m:
    pm.DensityDist('pot1', logp=cust_logp, shape=(2,))
    trace = pm.sample(1000)

pm.traceplot(trace);

pm.plot_posterior(trace);


I am working on the file "Distributions/DensityDist Example.ipynb".
The first task is:
Now can we do this from a histogram?
Can you write Python code for it?

dist = scipy.stats.rayleigh(loc=2, scale=20)
rvs = dist.rvs(10000)
sns.distplot(rvs)

# need to make a callable function of the histogram of this.
h, b = np.histogram(rvs, 30)
b = tb.bin_edges_to_center(b)
plt.plot(b, h)

f = scipy.interpolate.interp1d(b,h, bounds_error=False, fill_value=0)
XX = np.linspace(0, 100, 1000)
plt.plot(XX, f(XX))

from theano.tensor import _shared

# @theano.as_op(itypes=[theano.tensor.dscalar],otypes=[theano.tensor.dscalar])
def dens_fn(x):
    return tt.log(f(x) )
# dens_fn(_shared(np.linspace(-1, 5, 10)))

with pm.Model() as model:
    dens = pm.DensityDist('density', logp=dens_fn)
    trace = pm.sample(5000)

pm.traceplot(trace);

pm.plot_posterior(trace);


I am working on the file "Distributions/fatiguelife.ipynb".
The first task is:
Do this in pymc3
Can you write Python code for it?

c=5
def fatiguelife_pdf_dist(x, c=5):
    ans = tt.log((tt.abs_(x)+1) / (2*c*tt.sqrt(2*np.pi*tt.abs_(x)**3)) *
                 tt.exp(-(tt.abs_(x)-1)**2/(2*tt.abs_(x)*c**2)))
    return ans

with mc.Model() as model:
    fatiguelife = mc.DensityDist('fatiguelife', logp=fatiguelife_pdf_dist, testval=2)

    trace = mc.sample(2000, njobs=2)

trace['fatiguelife'][trace['fatiguelife']>0]
plt.hist(trace['fatiguelife'][trace['fatiguelife']>0], 100, normed=True, histtype='step');
plt.yscale('log')

x = tb.linspace(0, 400, 1000)
plt.loglog(x, fatiguelife_pdf(x, 5), c='r')

traceplot(trace, combined=True)

tt.abs_

mc.DensityDist?

plt.hist?


I am working on the file "Do it yourself.ipynb".
The first task is:
Data - Titanic data set
From a sample of the RMS Titanic data, we can see the various features present for each passenger on the ship:
* Survived: Outcome of survival (0 = No; 1 = Yes)
* Pclass: Socio-economic class (1 = Upper class; 2 = Middle class; 3 = Lower class)
* Name: Name of passenger
* Sex: Sex of the passenger
* Age: Age of the passenger (Some entries contain NaN)
* SibSp: Number of siblings and spouses of the passenger aboard
* Parch: Number of parents and children of the passenger aboard
* Ticket: Ticket number of the passenger
* Fare: Fare paid by the passenger
* Cabin: Cabin number of the passenger (Some entries contain NaN)
* Embarked: Port of embarkation of the passenger (C = Cherbourg; Q = Queenstown; S = Southampton)
More about this data set can be found on Kaggle website.
Can you write Python code for it?

# TODO import proper Python libraries to examine and investigate Titanic data set
import matplotlib.pyplot as plt

# TODO load Titanic training set. File name is titanic_train.csv
import pandas as pd
df = pd.read_csv('titanic_train.csv')


I am working on the file "Do it yourself.ipynb".
The first task is:
Data investigation
Can you write Python code for it?

# TODO Check does any column in Titanic data set contains NaN values

# TODO Plot Pclass and Fare data distribution

# TODO Count how many passangers are over 40 years

# TODO Count how many men among passangers are over 40 years

# TODO Count how many men among passangers are over 40 years survived

# TODO Plot data distribution

# TODO if children are considered to be under the age of 16, how many children were in Titanic

# TODO How many men named Edward were among the passangers

# TODO experiment yourself a bit ;-)


I am working on the file "Do it yourself.ipynb".
The first task is:
Most basic Python string functions are:
1. len() - checks length of the string
usage example:
python
len(string)
2. lower() - creates lower case string
usage example:
python
string.lower()
3. upper() - creates upper case string
usage example:
python
string.upper()
4. str() - creates string / implicit string conversion
usage example:
python
str(string)
Can you write Python code for it?

# TODO Check what above given functions will produce from following variables:
a = 'Some test string...'
b = 'WE ARE LEARNING...'
c = 123

# TODO Concatenate all variables a, b and c into one and print it out

# TODO String formatting, usefull for logging and debugging
print "The %s who %s %s!" % ("Knights", "say", "Ni")

string1 = ' embedded string '
string2 = ' This is one string {}'.format(string1)
string2

import random
# TODO print the biggest number from the three given bellow num1, num2 and num3
num1 = random.randint(1, 100)
num2 = random.randint(1, 100)
num3 = random.randint(1, 100)

# TODO if number1 is bigger, print "number1 is bigger"
# if number2 is bigger, print "number2 is bigger"
# if they are equal, print "Numbers are equal, you had 1% chance to get this..."
number1 = random.randint(1, 100)
number2 = random.randint(1, 100)

# TODO if you are German and n is greater than m
# print upper case lc variable otherwise
# print lower case up variable
n = random.randint(1, 100)
m = random.randint(1, 100)
german = ? (True/False)
lc = 'lower case string'
up = 'UPPER CASE STRING'

# TODO remove false Beatle form the list
beatles = ["john","paul","george","ringo","stuart"]

# TODO print out all the beatles with the loop

# TODO make John and Ringo switch their places in the list

# TODO So as a reminder the Beatles are John Lennon, Paul McCartney, George Harrison and Ringo Starr
# in that respect attach proper last name to every Beatle in the list

# Now just execute this...
%run man.py


I am working on the file "Doc/Notebooks/GephiStreaming_UserGuide.ipynb".
The first task is:
... and export it directly export it into the active gephi workspace. After executing the following code, the graph should be available in the first gephi workspace. Attention: Any graph currently contained in that workspace will be overriden.
Can you write Python code for it?

client = gephi.streaming.GephiStreamingClient()
client.exportGraph(G)


I am working on the file "Doc/Notebooks/GephiStreaming_UserGuide.ipynb".
The first task is:
Changing the server URL
By default, the streaming client in NetworKit connects to http://localhost:8080/workspace0, i.e. the first workspace of the local gephi instance. One might want to connect to a gephi instance running on a remote host or change the used port (this can be done in gephi by selecting Settings within the Streaming tab). To change the url in NetworKit, simply pass it upon the creation of the client object:
Can you write Python code for it?


I am working on the file "Doc/Notebooks/GephiStreaming_UserGuide.ipynb".
The first task is:
Exporting node values
We now apply a community detection algorithm to the generated random graph and export the community as a node attribute to gephi. Any python list or Partition object can be exported. Please note that only the attribute itself is transfered, so make sure you called exportGraph(graph) first.
Can you write Python code for it?

communities = community.detectCommunities(G)

client.exportNodeValues(G, communities, "community")


I am working on the file "Doc/Notebooks/GephiStreaming_UserGuide.ipynb".
The first task is:
Introduction
NetworKit provides an easy interface to Gephi that uses the Gephi graph streaming plugin. To be able to use it, install the Graph Streaming plugin using the Gephi plugin manager. Afterwards, open the Streaming window by selecting Windows/Streaming in the menu.
Workflow
Once the plugin is installed in gephi, create a new project and start the Master Server in the Streaming tab within gephi. The running server will be indicated by a green dot.
As an example, we generate a random graph...
Can you write Python code for it?

G = generators.ErdosRenyiGenerator(300, 0.2).generate()
G.addEdge(0, 1) #We want to make sure this specific edge exists, for usage in an example later.


I am working on the file "Doc/Notebooks/GephiStreaming_UserGuide.ipynb".
The first task is:
The node attribute can now be selected and used in gephi, for partitioning or any other desired scenario.
Exporting edges scores
Just like node values, we can export edge values. After graph creation, each edge is assigned an integer id that is then used to access arbitrary attribute vectors, so any python list can be exported to gephi. In the following example, we assign an even number edge and export that score to gephi.
Can you write Python code for it?

edgeScore = [2*x for x in range(0, G.upperEdgeIdBound())]
client.exportEdgeValues(G, edgeScore, "myEdgeScore")


I am working on the file "Doc/Notebooks/Link_Prediction.ipynb".
The first task is:
2) Helper function to draw ROC curve in the notebook
Can you write Python code for it?

def drawROC(indexName, fpr, tpr):
    plt.suptitle(indexName, fontsize=18, fontweight="bold")
    plt.xticks([z / 10 for z in range(11)])
    plt.yticks([z / 10 for z in range(11)])
    plt.xlabel("False positive rate", fontsize=18)
    plt.ylabel("True positive rate", fontsize=18)
    plt.axis([0, 1, 0, 1])
    curve, = plt.plot(fpr, tpr)
    plt.plot([0, 1], [0, 1], 'grey', ls='--')
def drawUnsupervisedROC(indexName, predictions):
    fpr, tpr = roc.getCurve(predictions)
    drawROC(indexName, fpr, tpr)
def drawSupervisedROC(indexName, predictions, testLabels):
    predictionLabels = [p[1] for p in predictions]
    fpr, tpr, thresholds = metrics.roc_curve(testLabels, predictionLabels)
    drawROC(indexName, fpr, tpr)


I am working on the file "Doc/Notebooks/Link_Prediction.ipynb".
The first task is:
4) Initialize ROC/PR metrics for testGraph
Can you write Python code for it?

roc = lp.ROCMetric(testGraph)
pr = lp.PrecisionRecallMetric(testGraph)


I am working on the file "Doc/Notebooks/Link_Prediction.ipynb".
The first task is:
5) Generate training and feature graphs
Can you write Python code for it?

trainingGraph = lp.RandomLinkSampler.byPercentage(testGraph, 0.9)
print("Left out", testGraph.numberOfEdges() - trainingGraph.numberOfEdges(), "edges from test graph")
featureGraph = lp.RandomLinkSampler.byPercentage(trainingGraph, 0.7)
print("Left out", trainingGraph.numberOfEdges() - featureGraph.numberOfEdges(), "edges from training graph")


I am working on the file "Doc/Notebooks/Link_Prediction.ipynb".
The first task is:
6) Initialize similarity indices
Can you write Python code for it?

featureCommonNeighborsIndex = lp.CommonNeighborsIndex(featureGraph)
featureJaccardIndex = lp.JaccardIndex(featureGraph)
featurePreferentialAttachmentIndex = lp.PreferentialAttachmentIndex(featureGraph)
featureAdamicAdarIndex = lp.AdamicAdarIndex(featureGraph)
featureUDegreeIndex = lp.UDegreeIndex(featureGraph)
featureVDegreeIndex = lp.VDegreeIndex(featureGraph)
featureAlgebraicDistanceIndex = lp.AlgebraicDistanceIndex(featureGraph, 5, 15)
featureAlgebraicDistanceIndex.preprocess()
featureNeighborhoodDistanceIndex = lp.NeighborhoodDistanceIndex(featureGraph)
featureTotalNeighborsIndex = lp.TotalNeighborsIndex(featureGraph)
featureNeighborsMeasureIndex = lp.NeighborsMeasureIndex(featureGraph)
featureSameCommunityIndex = lp.SameCommunityIndex(featureGraph)
featureAdjustedRandIndex = lp.AdjustedRandIndex(featureGraph)
featureResourceAllocationIndex = lp.ResourceAllocationIndex(featureGraph)

commonNeighborsIndex = lp.CommonNeighborsIndex(trainingGraph)
jaccardIndex = lp.JaccardIndex(trainingGraph)
preferentialAttachmentIndex = lp.PreferentialAttachmentIndex(trainingGraph)
adamicAdarIndex = lp.AdamicAdarIndex(trainingGraph)
uDegreeIndex = lp.UDegreeIndex(trainingGraph)
vDegreeIndex = lp.VDegreeIndex(trainingGraph)
algebraicDistanceIndex = lp.AlgebraicDistanceIndex(trainingGraph, 5, 15)
algebraicDistanceIndex.preprocess()
neighborhoodDistanceIndex = lp.NeighborhoodDistanceIndex(trainingGraph)
totalNeighborsIndex = lp.TotalNeighborsIndex(trainingGraph)
neighborsMeasureIndex = lp.NeighborsMeasureIndex(trainingGraph)
sameCommunityIndex = lp.SameCommunityIndex(trainingGraph)
adjustedRandIndex = lp.AdjustedRandIndex(trainingGraph)
resourceAllocationIndex = lp.ResourceAllocationIndex(trainingGraph)

featureLinkPredictors = (featureCommonNeighborsIndex, featureJaccardIndex, featureAdamicAdarIndex, featurePreferentialAttachmentIndex, featureUDegreeIndex, featureVDegreeIndex, featureAlgebraicDistanceIndex, featureNeighborhoodDistanceIndex, featureTotalNeighborsIndex, featureNeighborsMeasureIndex, featureSameCommunityIndex,...


I am working on the file "Doc/Notebooks/Link_Prediction.ipynb".
The first task is:
7) Generate testing and training sets
Can you write Python code for it?

trainingSet = lp.MissingLinksFinder(featureGraph).findAtDistance(2)
testingSet = lp.MissingLinksFinder(trainingGraph).findAtDistance(2)

labelTrain = lp.getLabels(trainingSet, trainingGraph)
featureTrain = lp.getFeatures(trainingSet, *featureLinkPredictors)

labelTest = lp.getLabels(testingSet, testGraph)
featureTest = lp.getFeatures(testingSet, *linkPredictors)

print("Training set size:", len(trainingSet))
print("Testing set size:", len(testingSet))

# Create and train classifier
dtc = ensemble.BaggingClassifier(n_estimators=25, max_features=0.2, n_jobs=-1)
dtc.fit(featureTrain, labelTrain)
# Actual prediction
predictions = dtc.predict_proba(featureTest)
drawSupervisedROC("DT Bagging", predictions, labelTest)

cnPreds = commonNeighborsIndex.runOn(testingSet)
drawUnsupervisedROC("Common Neighbors", cnPreds)

lp.LinkThresholder.byCount(cnPreds, 10)


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_1.ipynb".
The first task is:
Another very useful feature is to determine if an edge is present and what its weight is. In case of unweighted graphs, edges have the default weight 1.
Can you write Python code for it?

edgeExists = G.hasEdge(42,11)
if edgeExists:
    print("Weight of existing edge:", G.weight(42,11))
print("Weight of nonexisting edge:", G.weight(42,12))


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_1.ipynb".
The first task is:
In case a Python warning appears that recommends an update to Python 3.4, simply ignore it for this tutorial. Python 3.3 works just as fine for our purposes.
IPython lets us use familiar shell commands in a Python interpreter. Use one of them now to change into the main directory of your NetworKit installation:
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_1.ipynb".
The first task is:
In the course of this tutorial, we are going to work (among others) on the PGPgiantcompo network, a social network/web of trust in which nodes are PGP keys and an edge represents a signature from one key on another (web of trust). It is distributed with NetworKit as a good starting point.
The Graph Object
Graph is the central class of NetworKit. An object of this type represents an optionally weighted network. In this tutorial we work with undirected graphs, but NetworKit supports directed graphs as well.
Let us inspect several of the methods which the class provides. Maybe the most basic information is the number of nodes and edges in as well as the name of the network.
Can you write Python code for it?

n = G.numberOfNodes()
m = G.numberOfEdges()
print(n, m)

G.toString()


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_1.ipynb".
The first task is:
Interesting Features of a Network
Let us become more concrete: In the talk that accompanies this tutorial you learned about basic network features. Go back to the 'Analytics' section of the slides and answer the following questions within the box below, including the code which found your answer (click on the box to enter text). If you need information on method prototypes, you have at least two options: Use the built-in code completion (tab) or the project website, which offers documentation in the form of an automatically generated reference: https://networkit.iti.kit.edu/documentation/ (Python/C++ Documentation in the left navigation bar).
After you answered the questions, go on with Tutorial #2.
Q&A Session #1

Who (which vertex) has the least/most 'friends' and how many?
Answer:

How many neighbors does a vertex have on average?
Answer:

Does the degree distribution follow a power law?
Answer:

How often is the friend of a friend also a friend? Let's go for the average fraction here, other definitions are possible...
Answer:

How many connected components does the graph have?
Answer:
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_1.ipynb".
The first task is:
Many graph algorithms can be expressed with iterators over nodes or edges. As an example, let us iterate over the nodes to determine how many of them have more than 100 neighbors.
Can you write Python code for it?

count = 0 # counts number of nodes with more than 100 neighbors
for v in G.nodes():
    if G.degree(v) > 100:
        count = count + 1
print("Number of nodes with more than 100 neighbors: ", count)


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_1.ipynb".
The first task is:
NetworKit stores nodes simply as integer indices. Edges are pairs of such indices. The following prints the indices of the first ten nodes and edges, respectively.
Can you write Python code for it?

V = G.nodes()
print(V[:10])
E = G.edges()
print(E[:10])


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_1.ipynb".
The first task is:
Reading Graphs
Let us start by reading a network from a file on disk: PGPgiantcompo.graph. NetworKit supports a number of popular graph file formats, among them the METIS adjacency list format. There is a convenient function in the top namespace to read a graph from a file:
Can you write Python code for it?

G = readGraph("input/PGPgiantcompo.graph", Format.METIS)


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_2.ipynb".
The first task is:
Differences between network types
As indicated by the previous task, graph theory and graph algorithms have been research foci for quite some time. In comparison, complex networks and their analysis have become a focus of investigation only recently. The reason for our interest in complex networks is their similarity to many real-world phenomena such as social interactions, web graphs, food webs, protein interactions and so forth.
But what makes a network complex or not complex?
Well, complex networks have 'non-trivial' topological features. Let us explore this statement in more detail with the help of data. To this end, we look at a social network, a technical mesh and an Erdös-Renyi random graph.
Can you write Python code for it?

# Load/generate 3 graphs of different types
mit8 = readGraph("input/MIT8.edgelist", Format.EdgeListTabZero)
airf1 = readGraph("input/airfoil1.graph", Format.METIS)
gen = generators.ErdosRenyiGenerator(1000, 0.01)
er1000 = gen.generate()


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_2.ipynb".
The first task is:
Eulerian Cycles
Before we look at different network types, let us reflect on the distant roots of network analysis. Graph theory is one of its ancestors and has been around for several centuries now. As an example, think of Euler's problem from 1736 of finding a 'Eulerian tour/cycle' over the Pregel bridges in Königsberg.
Q&A session #2

Do you remember how the problem of deciding wheter a graph has a Eulerian cycle can be solved for arbitrary undirected graphs? If not, use the web to find helpful characterizations of graphs with Eulerian cycles. Write it down here.
Answer:

Then enter code below to decide whether a graph has such a cycle or not.
Insert code in next cell

Test it on the PGP graph! What is the result? Does the result meet your expectations?
Answer:
Can you write Python code for it?

# 2-2) and 2-3) Decide whether graph is Eulerian or not


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_2.ipynb".
The first task is:
Some context on these networks is given below, first for MIT8. It stems from a larger collection of Facebook networks from the early days of the online social network. MI8 models

"Facebook friendships at 100 US universities at some time in 2005, as well as a number of node attributes such as dorm, gender, graduation year, and academic major.  The data was apparently provided directly by Facebook. (...) It does not include the names of individual or even of any of the node attributes (they have been given integer ids)." http://sociograph.blogspot.de/2011/03/facebook100-data-and-parser-for-it.html

The airfoil1 graph is a mesh that stems from a two-dimensional numerical simulation where one is interested in the airflow around an airplane wing. Meshes are usually easy to visualize, two-dimensional meshes are even planar in most (if not all reasonable) cases. The picture below illustrates this. The colors signify different vertex blocks, which we can ignore here.
Can you write Python code for it?

from IPython.core.display import Image
Image('input/airfoil1-10p.png')


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_3.ipynb".
The first task is:
Centrality Measures
The $k$-core decomposition is rather, as the name suggests, a decomposition of the vertices into discrete subsets. Nodes with the same coreness (i.e. in the same shell) have equal importance. Rankings where many vertices are equally important are often not very meaningful. That is why the $k$-core decomposition should not be interpreted as a fine-grained ranking mechanism.
Q&A Session #6

Take the Facebook graph MIT8 and find the most central nodes. Take the relevance of their neighbors into account. Consider that MIT8 models a social network, not a web graph. Which algorithm would you choose? (Hint: Look at the lecture slides!)
Answer:

What are the 15 most important nodes according to the method in 1)?
Answer:

What other centrality measures do you recall?
Answer:

After you answered the questions, proceed with Tutorial #4.
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_3.ipynb".
The first task is:
Thus, ... % of the users in this network have fewer friends than their friends have on average. While this result cannot be generalized exactly like this to other networks, the qualitative effect is often seen in social (and other) networks. Thus, let us now consider measures that broaden the rather narrow scope of the degree.
$k$-core Decomposition
Thus, the next concept we consider is the $k$-core decomposition. To answer the following Q&A session, go back to the lecture slides.
Q&A Session #5

What is the definition of an $i$-core? (Note that $k$ is often used for the largest core only!)
Answer:

Why do you think it can be considered a more robust measure for importance compared to the degree?
Answer:

Compute the $k$-core decomposition of the three networks used before. Then print the non-empty $i$-shells by using the method coreNumbers(). What results (similarities/differences) do you expect? Are these expectations met by the results?
Answer:

What disadvantage do you see when using $k$-core decomposition to rate nodes according to their importance?
Answer:
Can you write Python code for it?

# Code for 5-3)
mit8 = readGraph("input/MIT8.edgelist", Format.EdgeListTabZero)
airf1 = readGraph("input/airfoil1.graph", Format.METIS)
gen = generators.ErdosRenyiGenerator(1000, 0.01)
er1000 = gen.generate()

# for g in {mit8, airf1, er1000}:


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_4.ipynb".
The first task is:
Community Detection
This section demonstrates the community detection capabilities of NetworKit. Community detection is concerned with identifying groups of nodes which are significantly more densely connected to each other than to the rest of the network.
Code for community detection is contained in the community module. The module provides a top-level function to quickly perform community detection with a suitable algorithm and print some stats about the result.
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_Tutorial_Part_4.ipynb".
The first task is:
The visualization module, which is based on external code for graph drawing, provides a function which visualizes the community graph for a community detection solution: Communities are contracted into single nodes whose size corresponds to the community size. For problems with hundreds or thousands of communities, this may take a while.
Can you write Python code for it?

communities = _
viztasks.drawCommunityGraph(G, communities)


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Approximation of Betweenness
Since exact calculation of betweenness scores is often out of reach, NetworKit provides an approximation algorithm based on path sampling. Here we estimate betweenness centrality in PGPgiantcompo, with a probabilistic guarantee that the error is no larger than an additive constant $\epsilon$.
Can you write Python code for it?

abc = centrality.ApproxBetweenness(G, epsilon=0.1)
abc.run()


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Centrality
Centrality measures the relative importance of a node within a graph. Code for centrality analysis is grouped into the centrality module.
Betweenness Centrality
We implement Brandes' algorithm for the exact calculation of betweenness centrality. While the algorithm is efficient, it still needs to calculate shortest paths between all pairs of nodes, so its scalability is limited. We demonstrate it here on the small Karate club graph.
Can you write Python code for it?

K = readGraph("input/karate.graph", Format.METIS)

bc = centrality.Betweenness(K)
bc.run()


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Choice of Algorithm
The community detection function used a good default choice for an algorithm: PLM, our parallel implementation of the well-known Louvain method. It yields a high-quality solution at reasonably fast running times. Let us now apply a variation of this algorithm.
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Community Detection
This section demonstrates the community detection capabilities of NetworKit. Community detection is concerned with identifying groups of nodes which are significantly more densely connected to eachother than to the rest of the network.
Code for community detection is contained in the community module. The module provides a top-level function to quickly perform community detection with a suitable algorithm and print some stats about the result.
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Connected Components
A connected component is a set of nodes in which each pair of nodes is connected by a path. The following function determines the connected components of a graph:
Can you write Python code for it?

cc = components.ConnectedComponents(G)
cc.run()
print("number of components ", cc.numberOfComponents())
v = 0
print("component of node ", v , ": " , cc.componentOfNode(0))
#print("map of component sizes: ", cc.getComponentSizes())


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Core Decomposition
A $k$-core decomposition of a graph is performed by successicely peeling away nodes with degree less than $k$. The remaining nodes form the $k$-core of the graph.
Can you write Python code for it?

K = readGraph("input/karate.graph",Format.METIS)
coreDec = centrality.CoreDecomposition(K)
coreDec.run()


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Eigenvector Centrality and PageRank
Eigenvector centrality and its variant PageRank assign relative importance to nodes according to their connections, incorporating the idea that edges to high-scoring nodes contribute more. PageRank is a version of eigenvector centrality which introduces a damping factor, modeling a random web surfer which at some point stops following links and jumps to a random page. In PageRank theory, centrality is understood as the probability of such a web surfer to arrive on a certain page. Our implementation of both measures is based on parallel power iteration, a relatively simple eigensolver.
Can you write Python code for it?

# Eigenvector centrality
ec = centrality.EigenvectorCentrality(K)
ec.run()
ec.ranking()[:10] # the 10 most central nodes

# PageRank
pr = centrality.PageRank(K, 1e-6)
pr.run()
pr.ranking()[:10] # the 10 most central nodes


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Modularity is the primary measure for the quality of a community detection solution. The value is in the range [-0.5,1] and usually depends both on the performance of the algorithm and the presence of distinctive community structures in the network.
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
NetworKit is a hybrid built from C++ and Python code: Its core functionality is implemented in C++ for performance reasons, and then wrapped for Python using the Cython toolchain. This allows us to expose high-performance parallel code as a normal Python module. On the surface, NetworKit is just that and can be imported accordingly:
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
NetworkX Compatibility
NetworkX is a popular Python package for network analysis. To let both packages complement eachother, and to enable the adaptation of existing NetworkX-based code, we support the conversion of the respective graph data structures.
Can you write Python code for it?

import networkx as nx
nxG = nxadapter.nk2nx(G) # convert from NetworKit.Graph to networkx.Graph
print(nx.degree_assortativity_coefficient(nxG))


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Search and Shortest Paths
A simple breadth-first search from a starting node can be performed as follows:
Can you write Python code for it?

v = 0
bfs = graph.BFS(G, v)
bfs.run()

bfsdist = bfs.getDistances()


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Settings
In this section we discuss global settings.
Logging
When using NetworKit from the command line, the verbosity of console output can be controlled via several loglevels, from least to most verbose: FATAL, ERROR, WARN, INFO, DEBUG and TRACE. (Currently, logging is only available on the console and not visible in the IPython Notebook).
Can you write Python code for it?

getLogLevel() # the default loglevel

setLogLevel("TRACE") # set to most verbose mode
setLogLevel("ERROR") # set back to default


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
Similarly, Dijkstra's algorithm yields shortest path distances from a starting node to all other nodes in a weighted graph. Because PGPgiantcompo is an unweighted graph, the result is the same here:
Can you write Python code for it?

dijkstra = graph.Dijkstra(G, v)
dijkstra.run()
spdist = dijkstra.getDistances()
sum(spdist) / len(spdist)


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
The 10 most central nodes according to betweenness are then
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
The Chung-Lu model (also called configuration model) generates a random graph which corresponds to a given degree sequence, i.e. has the same expected degree sequence. It can therefore be used to replicate some of the properties of a given real networks, while others are not retained, such as high clustering and the specific community structure.
Can you write Python code for it?

degreeSequence = [G.degree(v) for v in G.nodes()]
clgen = generators.ChungLuGenerator(degreeSequence)


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
The Graph Object
Graph is the central class of NetworKit. An object of this type represents an undirected, optionally weighted network. Let us inspect several of the methods which the class provides.
Can you write Python code for it?

n = G.numberOfNodes()
m = G.numberOfEdges()
print(n, m)

G.toString()


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
The contents of a partition object can be written to file in a simple format, in which each line i contains the subset id of node i.
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
The function prints some statistics and returns the partition object representing the communities in the network as an assignment of node to community label. Let's capture this result of the last function call.
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
The return value is a list of distances from v to other nodes - indexed by node id. For example, we can now calculate the mean distance from the starting node to all other nodes:
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
There is a large variety of formats for storing graph data in files. For NetworKit, the currently best supported format is the METIS adjacency format. Various example graphs in this format can be found here.  The readGraph function tries to be an intelligent wrapper for various reader classes. In this example, it uses the METISGraphReader which is located in the graphio submodule, alongside other readers. These classes can also be used explicitly:
Can you write Python code for it?

graphio.METISGraphReader().read("input/PGPgiantcompo.graph")
# is the same as: readGraph("input/PGPgiantcompo.graph", Format.METIS)


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
This network is unweighted, meaning that each edge has the default weight of 1.
Can you write Python code for it?


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
We have now calculated centrality values for the given graph, and can retrieve them either as an ordered ranking of nodes or as a list of values indexed by node id.
Can you write Python code for it?

bc.ranking()[:10] # the 10 most central nodes


I am working on the file "Doc/Notebooks/NetworKit_UserGuide.ipynb".
The first task is:
We have switched on refinement, and we can see how modularity is slightly improved. For a small network like this, this takes only marginally longer.
Visualizing the Result
We can easily plot the distribution of community sizes as follows. While the distribution is skewed, it does not seem to fit a power-law, as shown by a log-log plot.
Can you write Python code for it?

sizes = communities.subsetSizes()
sizes.sort(reverse=True)
ax1 = plt.subplot(2,1,1)
ax1.set_ylabel("size")
ax1.plot(sizes)

ax2 = plt.subplot(2,1,2)
ax2.set_xscale("log")
ax2.set_yscale("log")
ax2.set_ylabel("size")
ax2.plot(sizes)


I am working on the file "Doc/Notebooks/Sparsification.ipynb".
The first task is:
Example 4: Using scores, filters, and parameterization
In the following example, we illustrate possible usage of a sparsification score, a filter and a parameterization algorithm. Note that any of these might be exchanged.
Can you write Python code for it?

#Score calculation
sparsificationAlgorithm = sparsification.LocalDegreeSparsifier()
edgeScores = sparsificationAlgorithm.scores(G)

#Parameterization using binary search with up to 20 steps. The parameter value is in [0,1] and the size of the graph increases with increasing parameter value.
parameterization = sparsification.BinarySearchParameterization(True, 0.0, 1.0, 20)
parameter = parameterization.parameterize(sparsificationAlgorithm, G, edgeScores, 0.3) # We'd like to keep ~30% of edges

#Global filtering
globalFilter = sparsification.GlobalThresholdFilter(G, edgeScores, parameter, False) #Keep all edges with an edge attribute value below the given parameter
B = globalFilter.calculate()
B.size()


I am working on the file "Doc/Notebooks/Sparsification.ipynb".
The first task is:
Example 5: Using linearized scores with Gephi
An example where it might be desireable to work with 'linear' edge score is Filtering in Gephi. By 'linear' we mean that a value close to the mean value should yield a graph with about 50% of edges. The sparsification module offers a convenience function that offers this functionality and that can be applied to both node and edge scores.
Can you write Python code for it?

sparsificationAlgorithm = sparsification.LocalSimilaritySparsifier()
edgeScores = sparsificationAlgorithm.scores(G)
linearScores = sparsification.getRankAttribute(edgeScores)
gephiClient.exportEdgeValues(G, edgeScores, 'rank')


I am working on the file "Doc/Notebooks/Sparsification.ipynb".
The first task is:
For visualization purposes, let's also calculate and export the community structure:
Can you write Python code for it?

c = community.detectCommunities(G, community.PLM(G, refine=False, par='none'))
gephiClient.exportNodeValues(G, c, 'community')


I am working on the file "Doc/Notebooks/Sparsification.ipynb".
The first task is:
In order to obtain a reduced graph of a specific size, we can use the following convenience function which applies an appropriate parameterization algorithm to obtain a parameter value first:
Can you write Python code for it?

S = sparsificationAlgorithm.getSparsifiedGraphOfSize(G, 0.5)
S.size()


I am working on the file "Doc/Notebooks/Sparsification.ipynb".
The first task is:
In the following code piece, we need to specify a parameter value, which indirectly influences the size of the resulting graph.
Can you write Python code for it?

S = sparsificationAlgorithm.getSparsifiedGraph(G, 0.5)
S.size()


I am working on the file "Doc/Notebooks/Sparsification.ipynb".
The first task is:
Let's calculate an edge score for each of these definitions and export them all to Gephi:
Can you write Python code for it?

gephiClient = gephi.streaming.GephiStreamingClient()
gephiClient.clearGraph()
gephiClient.exportGraph(G)
for (algorithm, name) in sAlgorithms:
    edgeScores = algorithm.scores(G)
    linearEdgeScores = sparsification.getRankAttribute(edgeScores)
    gephiClient.exportEdgeValues(G, linearEdgeScores, name)


I am working on the file "Doc/Notebooks/Sparsification.ipynb".
The first task is:
Note that pre-calculated edge scores can be passed to these functions as well in order to avoid unnecessary recalculations.
Example 2: Exporting a sparsification score to Gephi
In order to filter edges within Gephi, we need to calculate the edge score:
Can you write Python code for it?

sparsificationAlgorithm = sparsification.LocalDegreeSparsifier()
edgeScores = sparsificationAlgorithm.scores(G)


I am working on the file "Doc/Notebooks/Sparsification.ipynb".
The first task is:
Sparsification algorithms that leave the set of nodes intact can be split up into edge score calculation and a global filtering step. All backbone algorithm implementations in the backbones-module are based on that insight. The module provides both low-level scores and filters and high-level convenience classes which can be identified by the suffix 'Backbone'.
Example 1: Simple calculation of a sparsified graph
Can you write Python code for it?


I am working on the file "Doc/Notebooks/Sparsification.ipynb".
The first task is:
We can now export it to Gephi (start the Gephi Streaming Master server first):
Can you write Python code for it?

gephiClient = gephi.streaming.GephiStreamingClient()
gephiClient.exportGraph(G)
gephiClient.exportEdgeValues(G, edgeScores, 'myAttribute')


I am working on the file "Doc/Notebooks/Tutorial_Solutions_Part_1.ipynb".
The first task is:
In the course of this tutorial, we are going to work (among others) on the PGPgiantcompo network, a social network/web of trust in which nodes are PGP keys and an edge represents a signature from one key on another (web of trust). It is distributed with NetworKit as a good starting point.
The Graph Object
Graph is the central class of NetworKit. An object of this type represents an optionally weighted network. In this tutorial we work with undirected graphs, but NetworKit supports directed graphs as well.
Let us inspect several of the methods which the class provides. Maybe the most basic information is the number of nodes and edges in as well as the name of the network.
Can you write Python code for it?


I am working on the file "Doc/Notebooks/Tutorial_Solutions_Part_1.ipynb".
The first task is:
Interesting Features of a Network
Let us become more concrete: In the talk that accompanies this tutorial you learned about basic network features. Go back to the 'Analytics' section of the slides and answer the following questions within the box below, including the code which found your answer (click on the box to enter text). If you need information on method prototypes, you have at least two options: Use the built-in code completion (tab) or the project website, which offers documentation in the form of an automatically generated reference: https://networkit.iti.kit.edu/documentation/ (Python/C++ Documentation in the left navigation bar).
After you answered the questions, go on with Tutorial #2.
Q&A Session #1

Who (which vertex) has the least/most 'friends' and how many?
Answer:

How many neighbors does a vertex have on average?
Answer:

Does the degree distribution follow a power law?
Answer:

How often is the friend of a friend also a friend? Let's go for the average fraction here, other definitions are possible...
Answer:

How many connected components does the graph have?
Answer:
Can you write Python code for it?

# Enter code for Q&A Session #1 here
argmin = 0
argmax = 0
mini = G.numberOfNodes()
maxi = 0
degsum = 0
for v in G.nodes():
    deg = G.degree(v)
    degsum = degsum + deg
    if deg < mini:
        mini = deg
        argmin = v
    if deg > maxi:
        maxi = deg
        argmax = v

avgdeg = degsum / G.numberOfNodes()

# Answers to 1-1) and 1-2)
print("1-1a) min: ", mini, " / argmin: ", argmin)
print("1-1b) max: ", maxi, " / argmax: ", argmax)
print("1-2)  avg: ", avgdeg)

# Answer to 1-3)
dd = sorted(centrality.DegreeCentrality(G).run().scores(), reverse=True)
import powerlaw
fit = powerlaw.Fit(dd)
print("1-3) Power law fit: ", fit.alpha)

# Answer to 1-4)
alcc = centrality.LocalClusteringCoefficient(G)
print("1-4)  Exact average local clustering coefficient: ", alcc)

# Answer to 1-5)
conncomp = components.ConnectedComponents(G)
conncomp.run()
print("1-5)  Number of components: ", conncomp.numberOfComponents())


I am working on the file "Doc/Notebooks/Tutorial_Solutions_Part_1.ipynb".
The first task is:
Many graph algorithms can be expressed with iterators over nodes or edges. As an example, let us iterate over the nodes to determine how many of them have more than 100 neighbors.
Can you write Python code for it?

count = 0 # counts number of nodes with more than 100 neighbors
for v in G.nodes():
    if G.degree(v) > 100:
        count = count + 1
print("Number of nodes with more than 100 neighbors: ", count)


I am working on the file "Doc/Notebooks/Tutorial_Solutions_Part_1.ipynb".
The first task is:
NetworKit stores nodes simply as integer indices. Edges are pairs of such indices. The following prints the indices of the first ten nodes and edges, respectively.
Can you write Python code for it?

V = G.nodes()
print(V[:10])
E = G.edges()
print(E[:10])


I am working on the file "Doc/Notebooks/Tutorial_Solutions_Part_1.ipynb".
The first task is:
Reading Graphs
Let us start by reading a network from a file on disk: PGPgiantcompo.graph. NetworKit supports a number of popular file formats. There is a convenient function in the top namespace which reads a graph from a file:
Can you write Python code for it?

G = readGraph("input/PGPgiantcompo.graph", Format.METIS)


I am working on the file "Doc/Notebooks/Tutorial_Solutions_Part_3.ipynb".
The first task is:
Thus, 86% of the users in this network have fewer friends than their friends have on average. While this result cannot be generalized exactly like this to other networks, the qualitative effect is often seen in social (and other) networks. Thus, let us now consider measures that broaden the rather narrow scope of the degree.
$k$-core Decomposition
Thus, the next concept we consider is the $k$-core decomposition. To answer the following Q&A session, go back to the lecture slides.
Q&A Session #5

What is the definition of an $i$-core? (Note that $k$ is often used for the largest core only!)
Answer:

Why do you think it can be considered a more robust measure for importance compared to the degree?
Answer:

Compute the $k$-core decomposition of the three networks used before. Then print the non-empty $i$-shells by using the method scores(). What results (similarities/differences) do you expect? Are these expectations met by the results?
Answer:

What disadvantage do you see when using $k$-core decomposition to rate nodes according to their importance?
Answer:
Can you write Python code for it?

# Code for 5-3)
mit8 = readGraph("input/MIT8.edgelist", Format.EdgeListTabZero)
airf1 = readGraph("input/airfoil1.graph", Format.METIS)
gen = generators.ErdosRenyiGenerator(1000, 0.01)
er1000 = gen.generate()

for g in {mit8, airf1, er1000}:
    print(g.toString())
    coreDec = centrality.CoreDecomposition(g)
    coreDec.run()
    print(set(coreDec.scores()))


I am working on the file "Doc/Notebooks/Tutorial_Solutions_Part_4.ipynb".
The first task is:
Community Detection
This section demonstrates the community detection capabilities of NetworKit. Community detection is concerned with identifying groups of nodes which are significantly more densely connected to each other than to the rest of the network.
Code for community detection is contained in the community module. The module provides a top-level function to quickly perform community detection with a suitable algorithm and print some stats about the result.
Can you write Python code for it?


I am working on the file "Doc/Notebooks/Tutorial_Solutions_Part_4.ipynb".
The first task is:
Q&A Session 8

Run PLMR as well. What are the main differences between the three algorithms PLM, PLMR, and PLP in terms of the solutions they compute and the time they need for this computation?
Answer:

Visualize the three results. Can you see aspects of your answer to 1) in the figure as well? Do the figures lead to other insights?
Answer:
Can you write Python code for it?

# Code for 8-1) and 8-2)
community.detectCommunities(G, algo=community.PLM(G,True))
communities = _
viztasks.drawCommunityGraph(G, communities)


I am working on the file "Doc/Notebooks/Tutorial_Solutions_Part_4.ipynb".
The first task is:
The visualization module, which is based on external code for graph drawing, provides a function which visualizes the community graph for a community detection solution: Communities are contracted into single nodes whose size corresponds to the community size. For problems with hundreds or thousands of communities, this may take a while.
Can you write Python code for it?

communities = _
viztasks.drawCommunityGraph(G, communities)


I am working on the file "Doc/uploads/docs/GephiStreaming_UserGuide.ipynb".
The first task is:
... and export it directly export it into the active gephi workspace. After executing the following code, the graph should be available in the first gephi workspace. Attention: Any graph currently contained in that workspace will be overriden.
Can you write Python code for it?

client = gephi.streaming.GephiStreamingClient(url='http://localhost:8080/workspace0')
client.exportGraph(G)


I am working on the file "Doc/uploads/docs/GephiStreaming_UserGuide.ipynb".
The first task is:
Exporting node values
We now apply a community detection algorithm to the generated random graph and export the community as a node attribute to gephi. Any python list or Partition object can be exported. Please note that only the attribute itself is transfered, so make sure you called exportGraph(graph) first.
Can you write Python code for it?

communities = community.detectCommunities(G)

client.exportNodeValues(G, communities, "community")


I am working on the file "Doc/uploads/docs/GephiStreaming_UserGuide.ipynb".
The first task is:
The node attribute can now be selected and used in gephi, for partitioning or any other desired scenario.
Exporting edges scores
Just like node values, we can export edge values. After graph creation, each edge is assigned an integer id that is then used to access arbitrary attribute vectors, so any python list can be exported to gephi. In the following example, we assign an even number edge and export that score to gephi.
Can you write Python code for it?

edgeScore = [2*x for x in range(0, G.upperEdgeIdBound())]
client.exportEdgeValues(G, edgeScore, "myEdgeScore")


I am working on the file "Doc/uploads/docs/SpectralCentrality.ipynb".
The first task is:
Compute the EVZ using the C++ backend and also display the 10 most important vertices, just as above. This should hopefully look similar...
Please note: The normalization argument may not be passed as a named argument to the C++-backed centrality measures. This is due to some limitation in the C++ wrapping code.
Can you write Python code for it?

evz = networkit.centrality.EigenvectorCentrality(G, True)
evz.run()
evz.ranking()[:10]


I am working on the file "Doc/uploads/docs/SpectralCentrality.ipynb".
The first task is:
First, we just compute the Python EVZ and display a sample. The "scores()" method returns a list of centrality scores in order of the vertices. Thus, what you see below are the (normalized, see the respective argument) centrality scores for G.nodes()[0], G.nodes()[1], ...
Can you write Python code for it?

evzSciPy = networkit.centrality.SciPyEVZ(G, normalized=True)
evzSciPy.run()
evzSciPy.scores()[:10]


I am working on the file "Doc/uploads/docs/SpectralCentrality.ipynb".
The first task is:
If everything went well, these should look similar, too.
Finally, we take a look at the relative differences between the computed centralities for the vertices:
Can you write Python code for it?

differences = [(max(x[0], x[1]) / min(x[0], x[1])) - 1 for x in zip(evz.scores(), evzSciPy.scores())]
print("Average relative difference: {}".format(sum(differences) / len(differences)))
print("Maximum relative difference: {}".format(max(differences)))


I am working on the file "Doc/uploads/docs/SpectralCentrality.ipynb".
The first task is:
Now, let's take a look at the PageRank. First, compute the PageRank using the C++ backend and display the 10 most important vertices. The second argument to the algorithm is the dampening factor, i.e. the probability that a random walk just stops at a vertex and instead teleports to some other vertex.
Can you write Python code for it?

pageRank = networkit.centrality.PageRank(G, 0.95, True)
pageRank.run()
pageRank.ranking()[:10]


I am working on the file "Doc/uploads/docs/SpectralCentrality.ipynb".
The first task is:
Same in Python...
Can you write Python code for it?

SciPyPageRank = networkit.centrality.SciPyPageRank(G, 0.95, normalized=True)
SciPyPageRank.run()
SciPyPageRank.ranking()[:10]


I am working on the file "Doc/uploads/docs/SpectralCentralityWithPandas.ipynb".
The first task is:
... then display it. What you will see is a list of the 10 most important vertices and their respective centralities according to the C++ / Python version of eigenvector centrality:
Can you write Python code for it?

rankTableEVZ = pd.DataFrame({"Python EVZ": evzSciPy.ranking()[:10], "C++ EVZ": evz.ranking()[:10]})
rankTableEVZ


I am working on the file "Doc/uploads/docs/SpectralCentralityWithPandas.ipynb".
The first task is:
Finally, we take a look at the relative differences between the computed centralities for the vertices:
Can you write Python code for it?


I am working on the file "Doc/uploads/docs/SpectralCentralityWithPandas.ipynb".
The first task is:
We now take a look at the 10 most central vertices according to the four heuristics. Here, the centrality algorithms offer the ranking() method that returns a list of (vertex, centrality) ordered by centrality. We first compute the remaining values...
Can you write Python code for it?

evz = networkit.centrality.EigenvectorCentrality(G, True)
evz.run()

pageRank = networkit.centrality.PageRank(G, 0.95)
pageRank.run()

pageRankSciPy = networkit.centrality.SciPyPageRank(G, 0.95, normalized=True)
pageRankSciPy.run()


I am working on the file "Doc2Vec_PopCorn.ipynb".
The first task is:
Function that labelize each review into 'TRAIN_i' , 'TEST_i' and 'UNSUP_i' where i is an index for each observation.
This procces is needed because in Doc2Vec it's necesary that each document, in this case
review, needs to have a label associated with it.
Can you write Python code for it?

def labelizeReviews(reviews, label_type):
    labelized = []
    for i,v in enumerate(reviews):
        label = '%s_%s'%(label_type,i)
        labelized.append(LabeledSentence(v, [label]))
    return labelized

x_concatenate = np.concatenate((x_train, x_unlabeled))

#Doing labilize procces to each data set, specifying in the parameter "label_type"
#what kind of data is the given data.
x_train2 = labelizeReviews(x_concatenate, 'TRAIN')
x_test2 = labelizeReviews(x_test, 'TEST')

size = 50
model= Doc2Vec(min_count=1, window=5, size=size, sample=1e-4, negative=5, workers=8)

#build vocab over all reviews
model.build_vocab(x_train2)

import random as ra

ra.seed(12345)
alpha = 0.025
min_alpha = 0.001
num_epochs = 5
alpha_delta = (alpha - min_alpha) / num_epochs

for epoch in range(num_epochs):
    print(epoch)
    ra.shuffle(x_train2)
    model.alpha = alpha
    model.min_alpha = alpha
    model.train(x_train2)
    alpha -= alpha_delta

#Infering over the test reviews in base the previous model built
test_vectors = np.zeros((len(x_test),size))
for i in range(len(x_test)):
    test_vectors[i] = model.infer_vector(x_test[i])

#Getting the train vectors from the model with the function docvecs.
#Remember that when we concatenate train and unlabeled sets, the first
#25,000 rows are from train set. The rest 50,000 rows are from the unlabeled set.
train_vectors = np.zeros((len(x_train), size))
for i in range(len(x_train)):
    train_vectors[i] = model.docvecs[i]

#model.save('my_model.doc2vec')


I am working on the file "Doc2Vec_PopCorn.ipynb".
The first task is:
Function to clean the data.
Removes html, special characters like punctuation and put all the words in lower case
Can you write Python code for it?

def review_to_wordlist( review, remove_stopwords=False ):
    # Function to convert a document to a sequence of words,
    # optionally removing stop words.  Returns a list of words.
    #
    clean_reviews = []
    for i in range(len(review)):
    # 1. Remove HTML
        review_text = BeautifulSoup(review[i]).get_text()
    #
    # 2. Remove non-letters
        review_text = re.sub("[^a-zA-Z]"," ", review_text)
    #
    # 3. Convert words to lower case and split them
        words = review_text.lower().split()
    #
    # 4. Optionally remove stop words (false by default)
        if remove_stopwords:
            stops = set(stopwords.words("english"))
            words = [w for w in words if not w in stops]
    #
        clean_reviews.append(words)
    # 5. Return a list of reviews
    return(clean_reviews)

#Cleaning the data with the reviews in each set of data
x_train = review_to_wordlist(x_train)
x_test = review_to_wordlist(x_test)
x_unlabeled = review_to_wordlist(x_unlabeled)


I am working on the file "Doc2Vec_PopCorn.ipynb".
The first task is:
Training and prediciting sentimient with Random Forest
Can you write Python code for it?

# Fit a random forest and extract predictions
forest = RandomForestClassifier(n_estimators = 100)
# Fitting the forest may take a few minutes
print "Fitting a random forest to labeled training data..."
forest = forest.fit(train_vectors,train["sentiment"])

results = forest.predict(test_vectors)

output = pd.DataFrame(data={"id":test["id"], "sentiment":results})
output.to_csv( "Doc2Vec.csv", index=False, quoting=3 )


I am working on the file "Doc2Vec_PopCorn.ipynb".
The first task is:
Training and predicting sentiment with Logistic Regression
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression

lgr = LogisticRegression()
lgr = lgr.fit(train_vectors, train["sentiment"])

results = lgr.predict(test_vectors)

output = pd.DataFrame(data={"id":test["id"], "sentiment":results})
output.to_csv( "Doc2Vec_lgr.csv", index=False, quoting=3 )


I am working on the file "Docker/mcstas/mcstasscript/McStasScript_demo.ipynb".
The first task is:
Plotting the returned data
The plot options looks at some metadata in the McStasData for plotting preferences. For this reason these options can be adjusted for individual data files instead of complex syntax for the plotting command.
Can you write Python code for it?

# Adjusting PSD_4PI plot
functions.name_plot_options("PSD_4PI", data, log=1, colormap="hot", orders_of_mag=5)

plot = plotter.make_sub_plot(data) # Making subplot of our monitors


I am working on the file "Docker_Basics.ipynb".
The first task is:
About Docker Image:
check http://imagelayers.io/ or https://microbadger.com/images/ for some further details of a specific image of layers.
build a docker image
we add some layers to the base image ubuntu:
```bash
Version: 0.0.1
FROM ubuntu
MAINTAINER CHWENG "CHWENG@example.com"
Avoid ERROR: invoke-rc.d: policy-rc.d denied execution of start.
RUN echo "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d
RUN apt-get update
RUN apt-get install -y nginx
RUN echo 'Hi, I am in the container' \
    >/var/www/html/index.html
    #>/usr/share/nginx/html/index.html
CMD [ "nginx", "-g", "daemon off;" ]EXPOSE 80
```
Can you write Python code for it?

%%bash
cd static_web/
ls
docker build -t "iiiedu/static_web" .

%%bash
docker images

%%bash
docker history iiiedu/static_web

%%bash
docker run -d -p 80 --name static_web iiiedu/static_web

%%bash
docker ps


I am working on the file "Docker_Basics.ipynb".
The first task is:
bash
docker run --rm tomcat:8.0 -P --name mytomcat
the flag --rm: remove the container after finishing the execusion
```bash
FROM tomcat:8.0
MAINTAINER iii education
USER root
RUN mkdir /usr/local/tomcat/webapps/jsptes
WORKDIR /usr/local/tomcat/webapps/jsptest
```
Can you write Python code for it?

docker run -d --name tomcat_jsp -p 8080:8080 -v $PWD/code:/usr/local/tomcat/webaps/jsptest iiiedu/tomcat


I am working on the file "Docker_Basics.ipynb".
The first task is:
docker attach
(attach to a running container)
bash
!docker attach daemon-dyson
docker inspect
inspect the selected container:
Can you write Python code for it?


I am working on the file "Docker_Basics.ipynb".
The first task is:
docker logs
Can you write Python code for it?

!docker logs daemon-dyson

!docker logs -t --tail 10 daemon-dyson

!docker logs -t -f daemon-dyson


I am working on the file "Docker_Basics.ipynb".
The first task is:
exercise: keep printing hello world before the termination of the daemon:
Can you write Python code for it?

!docker run --name daemon-dyson -d ubuntu /bin/bash -c "while true; do echo hello world;sleep 1;done"


I am working on the file "Docker_Basics.ipynb".
The first task is:
exercise: listen to the socket 9000
Can you write Python code for it?

%%bash

docker run -d --name java_sock -p 9000:9000 -v $PWD/code:/javatest iiiedu/socket 9000


I am working on the file "Docker_Basics.ipynb".
The first task is:
http://192.168.99.100:32768/
docker exec
use docker exec in order to access to an interactive terminal of the daemonized container (since if we use docker attach, we'll obtain a running terminal, which is not convenient if we'd like to do some extra things interactively.)
bash
docker exec -it static_web /bin/bash
to look for the location of the html file generated by nginx:
bash
root@7df500d5695b:/# find . -name "*.html"
./var/www/html/index.nginx-debian.html
./usr/share/nginx/html/index.html
./usr/share/doc/libfreetype6/ft2faq.html
./usr/share/doc/base-passwd/users-and-groups.html
./usr/share/doc/adduser/examples/adduser.local.conf.examples/skel.other/index.html
bash
docker exec -t stat /bin/ps
  PID TTY          TIME CMD
   35 ?        00:00:00 ps
use the flag -v(volume) to share files with the container:
bash
docker run -it -P --name web-apache -v ~/apache2/html:/var/www/html iiiedu/apache
(a volume that is going to share data with the container will be created)
exercise: run a nginx server
(the volume is mounted with the read only option)
bash
docker run -d -p 80:80 -v /tmp/www:/usr/share/nginx/html:ro nginx
troubleshooting: env in windows
In windows 7, I met some problem while trying to access docker from BASH (MINGW64). This is fixed after following
https://github.com/docker/docker/issues/22338
Can you write Python code for it?


I am working on the file "Docker_Basics.ipynb".
The first task is:
https://blog.dominodatalab.com/lesser-known-ways-of-using-notebooks/
Can you write Python code for it?

%%bash
cd ~/hadoop_base
vagrant package --output hadoop_base.box

%%bash
cd ~/hadoop_base
vagrant box add iiiedu/hadoop_base hadoop_base.box

%%bash
vagrant box list

%%bash
cd ~/
mkdir hadoop_node
cd hadoop_node
vagrant init iiiedu/hadoop_base

%%bash
cd ~/hadoop_node
vagrant up

%%bash
cd ~/hadoop_cluster
vagrant package master --output master.box
vagrant package slave1 --output slave1.box
vagrant package slave2 --output slave2.box
vagrant package slave3 --output slave3.box
vagrant package slave4 --output slave4.box
vagrant package slave5 --output slave5.box


I am working on the file "Docker_Basics.ipynb".
The first task is:
stop and remove a daemon:
Can you write Python code for it?

%%bash
docker stop daemon-dave
docker rm daemon-dave

%%bash
docker ps -a

!docker images


I am working on the file "Docker_Basics.ipynb".
The first task is:
the flag -f will keep refreshing the log file.
Can you write Python code for it?

!docker top daemon-dyson

showing the last three processes:

!docker ps -n 3


I am working on the file "Docker_Basics.ipynb".
The first task is:
tomcat
https://hub.docker.com/_/tomcat/
Can you write Python code for it?

%%bash
docker search tomcat

%%bash
docker pull tomcat:8.0


I am working on the file "Docker_Basics.ipynb".
The first task is:
workflow:
1. use box add to give vagrant an image (xxx.box) and give it a tag name, e.g. ubuntu/trusty64
2. vagrant init ubuntu/trusty64 (will create the file Vagrantfile)
3. vagrant up  (set-up the guest node according to the Vagrantfile and then turn on the guest node)
4. vagrant ssh (login to the guest node)

I was trying to find the folder which contains motd(message of the day) settings. This is done by using the forgotten "find" command:
bash
vagrant@vagrant-ubuntu-trusty-64:~$ sudo find / -type d -name "*motd*"
/etc/update-motd.d
check the file system and the harddisk:
```bash
root@vagrant-ubuntu-trusty-64:/vagrant# fdisk -l
Disk /dev/sda: 42.9 GB, 42949672960 bytes
4 heads, 32 sectors/track, 655360 cylinders, total 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00021928
Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048    83886079    41942016   83  Linux
root@vagrant-ubuntu-trusty-64:/vagrant# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            241M   12K  241M   1% /dev
tmpfs            49M  344K   49M   1% /run
/dev/sda1        40G  1.4G   37G   4% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            245M     0  245M   0% /run/shm
none            100M     0  100M   0% /run/user
vagrant         201G  124G   77G  62% /vagrant
```
Indeed, the mounted mount point /vagrant is the directory that is shared in between the host and the guest.
Remark: use hostnamectl to modify the host name when necessary. I am not going to do this.
Can you write Python code for it?


I am working on the file "Docker_Notebooks_And_Reproducible_Research.ipynb".
The first task is:
Docker images are identified with hex string or tags
Interface is docker <subcommand>
docker push, docker pull, docker tag
docker export will create a archiveable tarball of an image's filesystem.
DockerHub is like GitHub

Installing
Here's what you need:

Linux kernel with control groups and namespaces
Support for a layered filesystem (like AUFS)
Docker Daemon / Server (written in Go)

|Linux

Ubuntu 14.04 or
See Docker installation instructions for distributions with Kernel 3.8 + later or
Kernel configuration instructions

Windows and Mac
Docker Machine

easy install of
Git Bash
VirtualBox
Lightweight Linux distribution

Docker

Mac OSX users can use the Docker client from the Mac bash shell

Comes with busybox shell -> Write your Docker build.sh and run.sh in Bourne shell

Docker Concepts
Image
A read-only file system layer

Can you write Python code for it?


I am working on the file "Docker_Notebooks_And_Reproducible_Research.ipynb".
The first task is:
The Problem
Even though computers are often considered deterministic, computational software is a rapidly evolving and changing landscape. Libraries are constantly adding new features and fixing issues.

Image source: http://www.michaelogawa.com/research/storylines/
Even libraries with the strictest backwards-compatibility policies can change in significant ways.

Image source: http://www.bonkersworld.net/backwards-compatibility/
A reproducible computational environment has a sufficiently consistent state for the computational task at hand.
For example, this can consist of

a similar CPU instruction set
libraries and executables available with a specific version and configuration options
a specific version of a given compiler
a specific version of a libc implementation
a specific version of the C++ standard library

Close But Not Good Enough
Source code
Does not include:

Compiler
Hardware it was built on
How it is configured
Package dependencies
Run-time environment
How to run it

Image source: https://www.youtube.com/watch?v=g1LgVfV5_ZQ
Package managers and distributions

There is not a consensus on the package manager
Packages become unsupported over time
What to do if a required library is not packaged?

Virtual machines (VMs)

Inefficient utilization of computational resources

Image source: http://time-az.com/images/2014/02/20140203carjam.jpg
Enter Linux Containers

Linux container systems , like Docker, are new type of tool to easily build, ship, and run reproducible, binary applications.
It is "good enough" for a reproducible computational environment.
In this talk, we will introduce Docker from the perspective a scientific research software engineer.  We will

Generate an understanding of what Docker is by comparing it to existing technologies.

Give an introduction to basic Docker concepts.

Describe how Docker fits...
Can you write Python code for it?

!docker run --rm busybox sh -c 'echo "Hello Docker World!"'


I am working on the file "Docker_Notebooks_And_Reproducible_Research.ipynb".
The first task is:
Volume
A directory within one or more containers that bypasses the Union File System

Data volumes are initialized when a container is created
Volumes can be shared and reused between containers
Changes to a data volume are made directly
Changes to a data volume will not be included when you update an image
Volume persist until no containers use them
Host directories can also be mounted as data volumes

Why use a data volume?

Store and share data
Expose data or code from the host to the Docker computational environment

Dockerfile
A sequence of instructions to generate a Docker image
Can you write Python code for it?

!mkdir -p docker-ls-data
!cp $PWD/Data/*.png docker-ls-data/

%%writefile docker-ls-data/Dockerfile

FROM busybox
MAINTAINER Matt McCormick <matt.mccormick@kitware.com>
RUN mkdir -p /Data
ADD *.png /Data/
VOLUME /Data
CMD ["/bin/sh", "-c", "ls /Data"]

!docker build -t ls-data ./docker-ls-data

!docker run --rm ls-data


I am working on the file "Docs/Notebooks/FingerprintGenerators.ipynb".
The first task is:
Currently available invariants generators are:
Can you write Python code for it?

atomInvariantsGen = rdFingerprintGenerator.GetAtomPairAtomInvGen()

generator = rdFingerprintGenerator.GetMorganGenerator(radius = 3, atomInvariantsGenerator = atomInvariantsGen)
fingerprint = generator.GetSparseCountFingerprint(mol)
non_zero = fingerprint.GetNonzeroElements()

print("Morgan with AtomPairAtomInvGen", non_zero)

atomInvariantsGen = rdFingerprintGenerator.GetMorganAtomInvGen()

generator = rdFingerprintGenerator.GetMorganGenerator(radius = 3, atomInvariantsGenerator = atomInvariantsGen)
fingerprint = generator.GetSparseCountFingerprint(mol)
non_zero = fingerprint.GetNonzeroElements()

# Default for Morgan FP
print("Morgan with MorganAtomInvGen", non_zero)

atomInvariantsGen = rdFingerprintGenerator.GetMorganFeatureAtomInvGen()

generator = rdFingerprintGenerator.GetMorganGenerator(radius = 3, atomInvariantsGenerator = atomInvariantsGen)
fingerprint = generator.GetSparseCountFingerprint(mol)
non_zero = fingerprint.GetNonzeroElements()

print("Morgan with MorganFeatureAtomInvGen", non_zero)

atomInvariantsGen = rdFingerprintGenerator.GetRDKitAtomInvGen()

generator = rdFingerprintGenerator.GetMorganGenerator(radius = 3, atomInvariantsGenerator = atomInvariantsGen)
fingerprint = generator.GetSparseCountFingerprint(mol)
non_zero = fingerprint.GetNonzeroElements()

print("Morgan with RDKitAtomInvGen", non_zero)

bondInvariantsGen = rdFingerprintGenerator.GetMorganBondInvGen()

generator = rdFingerprintGenerator.GetMorganGenerator(radius = 3, bondInvariantsGenerator = bondInvariantsGen)
fingerprint = generator.GetSparseCountFingerprint(mol)
non_zero = fingerprint.GetNonzeroElements()

# Default for Morgan FP
print("Morgan with MorganBondInvGen", non_zero)


I am working on the file "Docs/Notebooks/FingerprintGenerators.ipynb".
The first task is:
Custom Invariants
It is also possible to provide custom invariants instead of using a invariants generator
Can you write Python code for it?

generator = rdFingerprintGenerator.GetAtomPairGenerator()
fingerprint = generator.GetSparseCountFingerprint(simpleMol)
non_zero = fingerprint.GetNonzeroElements()

print(non_zero)

customAtomInvariants = [1, 1, 1]
fingerprint = generator.GetSparseCountFingerprint(simpleMol, customAtomInvariants = customAtomInvariants)
non_zero = fingerprint.GetNonzeroElements()

print(non_zero)


I am working on the file "Docs/Notebooks/FingerprintGenerators.ipynb".
The first task is:
Invariant generators
It is possible to use a custom invariant generators while creating fingerprints. Invariant generators provide values to be used as invariants for each atom or bond in the molecule and these values affect the generated fingerprint.
Can you write Python code for it?

simpleMol = Chem.MolFromSmiles('CCC')

generator = rdFingerprintGenerator.GetRDKitFPGenerator()
fingerprint = generator.GetSparseCountFingerprint(simpleMol)
non_zero = fingerprint.GetNonzeroElements()

print("RDKitFingerprint", non_zero)

atomInvariantsGen = rdFingerprintGenerator.GetAtomPairAtomInvGen()

generator = rdFingerprintGenerator.GetRDKitFPGenerator(atomInvariantsGenerator = atomInvariantsGen)
fingerprint = generator.GetSparseCountFingerprint(simpleMol)
non_zero = fingerprint.GetNonzeroElements()

print("RDKitFingerprint", non_zero)


I am working on the file "Docs/Notebooks/FingerprintGenerators.ipynb".
The first task is:
Types of fingerprint generators
Currently 4 fingerprint types are supported by fingerprint generators
Can you write Python code for it?

generator = rdFingerprintGenerator.GetAtomPairGenerator()
fingerprint = generator.GetSparseCountFingerprint(mol)
non_zero = fingerprint.GetNonzeroElements()

print("Atom pair", non_zero)

generator = rdFingerprintGenerator.GetMorganGenerator(radius = 3)
fingerprint = generator.GetSparseCountFingerprint(mol)
non_zero = fingerprint.GetNonzeroElements()

print("Morgan", non_zero)

generator = rdFingerprintGenerator.GetRDKitFPGenerator()
fingerprint = generator.GetSparseCountFingerprint(mol)
non_zero = fingerprint.GetNonzeroElements()

print("RDKitFingerprint", non_zero)

generator = rdFingerprintGenerator.GetTopologicalTorsionGenerator()
fingerprint = generator.GetSparseCountFingerprint(mol)
non_zero = fingerprint.GetNonzeroElements()

print("TopologicalTorsion", non_zero)


I am working on the file "Docs/Notebooks/MolStandardize.ipynb".
The first task is:
Or you can specify which subset of MolVSValidations you want when initializing
Can you write Python code for it?

validations = [rdMolStandardize.NoAtomValidation()]
vm = rdMolStandardize.MolVSValidation(validations)
mol = Chem.MolFromSmiles("COc1cccc(C=N[N-]C(N)=O)c1[O-].O.O.O.O=[U+2]=O")
vm.validate(mol)


I am working on the file "Docs/Notebooks/MolStandardize.ipynb".
The first task is:
The rdMolStandardize module contains the following convenience functions for quickly performing that standardization task:
-  rdMolStandardize.StandardizeSmiles()
-  rdMolStandardize.ValidateSmiles()
Other functions within the module:
-  rdMolStandardize.Cleanup()
-  rdMolStandardize.ChargeParent()
-  rdMolStandardize.FragmentParent()
rdMolStandardize.Cleanup() is equivalent to the molvs.standardize.Standardizer().standardize() function in molvs
The rdMolStandardize module contains the following classes, that allow you to develop your custom standardization process.
-  rdMolStandardize.CleanupParameters
-  rdMolStandardize.Normalize
-  rdMolStandardize.MetalDisconnector
-  rdMolStandardize.FragmentParent
-  rdMolStandardize.FragmentRemover
-  rdMolStandardize.Reionizer
-  rdMolStandardize.Uncharger
-  rdMolStandardize.RDKitValidation
-  rdMolStandardize.MolVSValidation
-  rdMolStandardize.AllowedAtomsValidation
-  rdMolStandardize.DisallowedAtomsValidation
TODO
-  rdMolStandardize.tautomer
rdMolStandardize.StandardizeSmiles()
The rdMolStandardize.StandardizeSmiles() convenience function contains all sensible default functionality to help get started
Can you write Python code for it?

sm = "[Na]OC(=O)c1ccc(C[S+2]([O-])([O-]))cc1"
rdMolStandardize.StandardizeSmiles(sm)


I am working on the file "Docs/Notebooks/MolStandardize.ipynb".
The first task is:
The rdMolStandardize.AllowedAtomsValidation class lets the user input a list of atoms, anything not on the list throws an error.
The rdMolStandardize.DisallowedAtomsValidation class also takes an input of a list of atoms and as long as there are no atoms from the list it is deemed acceptable.
Can you write Python code for it?

from rdkit.Chem.rdchem import Atom
atomic_no = [6,7,8]
allowed_atoms = [Atom(i) for i in atomic_no]
vm = rdMolStandardize.AllowedAtomsValidation(allowed_atoms)
mol = Chem.MolFromSmiles("CC(=O)CF")
vm.validate(mol)

atomic_no = [9, 17, 35]
disallowed_atoms = [Atom(i) for i in atomic_no]
vm = rdMolStandardize.DisallowedAtomsValidation(disallowed_atoms)
mol = Chem.MolFromSmiles("CC(=O)CF")
msg = vm.validate(mol)


I am working on the file "Docs/Notebooks/MolStandardize.ipynb".
The first task is:
rdMolStandardize validation methods
There are four different ways of validating molecules, each can by run with their different classes:
-  rdMolStandardize.RDKitValidation
-  rdMolStandardize.MolVSValidation
-  rdMolStandardize.AllowedAtomsValidation
-  rdMolStandardize.DisallowedAtomsValidation
All the classes have a validate() method to perform the validation.
The rdMolStandardize.RDKitValidation class validates the valency of every atom in the input molecule (much like RDKit automatically does).
Can you write Python code for it?

vm = rdMolStandardize.RDKitValidation()
mol = Chem.MolFromSmiles("CO(C)C", sanitize=False)
vm.validate(mol)


I am working on the file "Docs/Notebooks/MolStandardize.ipynb".
The first task is:
rdMolStandardize.Cleanup() function can take in an optional rdMolStandardize.CleanupParameters object which you can specify your own standardization reference files to use.
The member variables that you can change are:
-  CleanupParameters.acidbaseFile
-  CleanupParameters.normalizationsFile
-  CleanupParameters.fragmentFile
-  CleanupParameters.maxRestarts
-  CleanupParameters.preferOrganic
Can you write Python code for it?

params = rdMolStandardize.CleanupParameters()

print("Default acidbaseFile: %s" % params.acidbaseFile)
print("Default normalizationsFile: %s" % params.normalizationsFile)
print("Default fragmentFile: %s" % params.fragmentFile)
print("Default maxRestarts: %s" % params.maxRestarts)
print("Default preferOrganic: %s" % params.preferOrganic)