I am working on the file "T3/Pregunta4.ipynb".
The first task is:
Que se traduce a un accuracy de validación de 0.9991. Esto indica un posible overfit a los datos de validación, por lo que es de esperar un mal rendimiento al momento de probar los datos de testing.
Árbol de clasificación
Sin procesamiento
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier as Tree
from sklearn.metrics import accuracy_score
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import f1_score

def train_tree(depth, x_tr, y_tr, x_v, y_v):
    # Entrenar el árbol
    model = Tree()
    model.set_params(max_depth=depth, criterion='gini', splitter='best')
    model.fit(x_tr, y_tr)

    y_tr_pred = model.predict(x_tr)
    y_v_pred = model.predict(x_v)

    train_error = (accuracy_score(y_tr, y_tr_pred))
    test_error = (accuracy_score(y_v, y_v_pred))

    return (train_error, test_error)

def graph_tree_range(params, x_tr, y_tr, x_v, y_v):
    train_errors = []
    test_errors = []

    for depth in params:
        print('Trying depth %d' % depth)
        (train, test) = train_tree(depth, x_tr, y_tr, x_v, y_v)
        train_errors.append(train)
        test_errors.append(test)

    plt.figure(figsize=(10, 8))
    plt.plot(params, train_errors, label="Train accuracy")
    plt.plot(params, test_errors, label="Validation accuracy")
    plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
    plt.xlabel('Profundidad del árbol')
    plt.ylabel('Error')
    plt.show()

params = np.arange(1, 30, 1)
graph_tree_range(params, x_tr, y_tr, x_v, y_v)


I am working on the file "T3/Pregunta4.ipynb".
The first task is:
Se tiene que mientras más azul es el área, mayor es la dificultad de la red neuronal para clasificar correctamente las clases. Por tanto, se puede observar que a la red le cuesta más clasificar la letra 8 (h) considerando que es en dónde más se tiene conflictos al clasificar.
Cabe notar que es normal que exista una dificultad de clasificación en la diagonal debido a que es dificil distinguir una clase de si misma.
SVM no lineal
Sin procesamiento
La SVM no lineal sin procesamiento toma demasiado tiempo en converger a una solución, por lo que en lugar de buscar hiperparámetros correctos se utilizará una SVM no lineal con parámetro estándar C=0.5 y kernel tipo RBF para obtener algún resultado.
La búsqueda de parámetros se realizará en una SVM con procesamiento.
Can you write Python code for it?

from sklearn.svm import SVC as SVM
from sklearn.metrics import accuracy_score
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import f1_score

def train_svm(param):
    model = SVM()
    model.set_params(C=param, kernel='rbf')
    model.fit(x_tr, y_tr)

    y_tr_pred = model.predict(x_tr)
    y_v_pred = model.predict(x_v)

    train_error = (1-accuracy_score(y_tr, y_tr_pred))
    test_error = (1-accuracy_score(y_v, y_v_pred))

    return (train_error, test_error)

def graph_svm_range(params):
    train_errors = []
    test_errors = []

    for depth in params:
        print('Trying param %f' % depth)
        (train, test) = train_svm(depth)
        train_errors.append(train)
        test_errors.append(test)

    plt.figure(figsize=(10, 8))
    plt.plot(params, train_errors, label="Train Error")
    plt.plot(params, test_errors, label="Test Error")
    plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
    plt.xlabel('RBF')
    plt.ylabel('Error')
    plt.show()

train_svm(0.5)


I am working on the file "T3/Pregunta4.ipynb".
The first task is:
Selección de mejor modelo
Utilizando cada máquina con los mejores hiperparámetros y considerando datos con pre procesamiento, se tiene que el mejor modelo estará dado por el accuracy de los datos de testing. Se tiene:
Can you write Python code for it?

from sklearn.metrics import accuracy_score

# Neural network
y_t_pred = model3.predict_classes(x_t.values)
print("Accuracy red neuronal: %f" % accuracy_score(y_t, y_t_pred))

tree_accuracy = train_tree(30, x_tr_scaled, y_tr, x_t_scaled, y_t)
print("Accuracy decision tree: %f" % tree_accuracy[1])

x_t_dim = pca_model.transform(x_t_scaled)
svm_errors = train_svm(0.75, x_tr_dim, y_tr, x_t_dim, y_t)
svm_errors

svm_acc = (svm_errors[1] - 1)*-1
print("Accuracy SVM: %f" % svm_acc)


I am working on the file "T3/T3.3-Social-Minimal-Interaction.ipynb".
The first task is:
3. Fractal analysis
Despite of its apparent simplicity, the perceptual crossing paradigm comprises several embedded of dynamic interaction, resulting on auto-correlations of the signals over different time scales.
Critical systems typically display temporal and spatial scale invariance in the form of fractals and $1/f$ noise, reflecting the process of propagation of long-range interactions based on local effects. For the complex systems approach to cognitive science, self-organized criticallity is appealing because it allows us to imagine systems that are able to self-regulate coordinated behaviours at different scales in a distributed manner and without a central controller.
We argue that $1/f$ noise analysis can account not only for the integratedness the behaviour of an agencial system (e.g. the mental, psychological characteristics of human behaviour) but also can characterize the nature of social interaction process. In our experimental setup we have a broad range of kinds of social interaction: humans recognizing each others as such, humans interacting with bots with artificial behaviour, humans failing to recognize other humans, bots tricking humans... Can we characterize when genuine social interaction emerges? And if so, where does it lies?
For analyzing fractal exponents in the dynamics of social interaction we use the Detrended Fluctuation Analysis (DFA). Since the slope of the fluctuations in a logarithmic plot is not always linear for all scales, we check if there is any cutoff value in which a transition to a linear relation starts. We do this by searching for negative peaks in the second derivate of $F(n)$. We only do this on the right half of the values of n in the plot, in order to find only the cutoffs at larger scales. Once the cutoff is found, we analyze the slope of the function in the decade inferior to the cutoff value. In the cases where there is no cutoff value we analyze the interval $n \in [10^{-0.5},10^{0.5}]$.
3.1. Exercise
Run a DFA analysis to obtain the fractal index $\beta$.
-...
Can you write Python code for it?

def plot_dfa_perceptual(x, precision, title, drawPlot):
    ix = np.arange(np.log2(len(x)/4), 4, -precision)
    n = np.round(2**ix)
    [n, F] = dfa(x, L=n)
    n = n/115 # Time (seconds) = samples / sample_frequency
    indexes = (n>10**-0.5)&(n<10**0.5) # Time interval for calculating the slope
    P = np.polyfit(np.log(n[indexes]),np.log(F[indexes]), 1)
    beta = 2*P[0]-1 # beta=2*alpha-1
    if drawPlot:
        plt.title(title+r" ($\beta$ = {:0.2f})".format(beta))
        plt.xlabel('n')
        plt.ylabel('F(n)')
        plt.loglog(n, F)
        plt.loglog(n[indexes], np.power(n[indexes], P[0])*np.exp(P[1]), 'r')
    return [beta, n, F]

# Plot figures
series = vel_relative
plt.figure(figsize=(16, 4), dpi=72)
indexExamples = [60, 7, 11];
for i,ex in enumerate(indexExamples):
    x = series[ex,:]
    ax = plt.subplot(1,3,(i+1))
    plot_dfa_perceptual(x, 0.1, oppTypes[ex], True);


I am working on the file "T3/T3.3-Social-Minimal-Interaction.ipynb".
The first task is:
4. Interaction measures
We propose that genuine social interaction should be manifested by emerging integratedness in collective variables capturing the dynamics of this interactions. Concretely, we propose the changes in the distance between the two participants as a candidate variable for test this hypothesis. On the other hand, if social engagement truly arises from interaction dynamics, individual variables as the changes in the position of the agent or the opponent should not present significative changes in their levels of integratedness and thus the exponents obtained from 1/f analysis.
In order to analyze the interaction between the subjects, we take the time series of the distance between the two players (or the player and the bot agent). We compute the first derivative of the distance to obtain the variations in the distance i.e. whether the players are approaching or distancing themselves at each moment of time. Then we use a DFA algorithm [Peng et al. (2000)] to compute the correlations in the data series of distance variations.
Can you write Python code for it?

# Data
series = vel_relative

# Plot figures
plt.figure(figsize=(16, 4), dpi=72)
indexExamples = [60, 7, 11];
for i,ex in enumerate(indexExamples):
    ax = plt.subplot(1,3,(i+1))
    plot_dfa_perceptual(series[ex,:], 0.1, oppTypes[ex], True);


I am working on the file "T3/T3.3-Social-Minimal-Interaction.ipynb".
The first task is:
Now, we display the boxplot of the results to get an statistical overview. For the cases of the derivative of the player's position or the opponent's possition, we cannot assure an statistical difference between the distributions of β.
Can you write Python code for it?

# Calculate the average velocity of each serie
series = vel_player

betas = np.zeros(len(series));
for i in range(len(series)):
    [beta,_,_] = plot_dfa_perceptual(series[i,:], 0.5, oppTypes[i], False)
    betas[i] = beta

# Plot figures
plt.figure(figsize=(16, 4), dpi=72)
dataBox = [betas[indexOscill], betas[indexShadow], betas[indexHuman]]
plt.boxplot(dataBox);
plt.ylabel(r'$\beta$');
plt.xticks([1, 2, 3], ['Oscillatory', 'Shadow', 'Human']);


I am working on the file "T3/T3.3-Social-Minimal-Interaction.ipynb".
The first task is:
We can display the box-plot of the velocity to check if there are differences between groups.
- Try other velocity variables looking for differences between groups, e.g. velocity of opponent, relative velocity
Can you write Python code for it?

# Calculate the average velocity of each serie
vel_stats=np.mean(vel_relative,axis=1) # velocity of the player
#vel_stats=np.std(vel_opponent,axis=1) # velocity of the opponent
#vel_stats=np.std(vel_relative,axis=1) # relative velocity between player

# Plot figure
plt.figure(figsize=(16, 4), dpi=72)
dataBox = [vel_stats[indexOscill], vel_stats[indexShadow], vel_stats[indexHuman]]
plt.boxplot(dataBox);
plt.ylabel("Average velocity")
plt.xticks([1, 2, 3], ['Oscillatory', 'Shadow', 'Human']);


I am working on the file "T4/T4.5-Prisoner's-dilemma.ipynb".
The first task is:
Exercise 2
We can simulate a population of strategies, put them to play, and see how the population evolve. These are the rules:

There is a initial queue filled randomly with strategies passed as input.
It pulls the first 2 players of the queue and performs a game with 50 rounds per game.
If one player reaches 70 points, 2 players with the same strategy will be added to the queue. At that time, if the other one has more than 50 points, 1 player with the same strategy will be added to the queue. Both scores reset.
Repeat until queue is empty or number of generations limit has reached.

Lets do some tests with 200 generations each, and try to get the best strategies.
Can you write Python code for it?

from pydilemma.generational import *

Generational('PrisonerMatrix', ['TitForTat', 'Alternate', 'Majority', 'Naive', 'Nice'], 200).start()


I am working on the file "TB_01/run.ipynb".
The first task is:
After having setup the Hamilton, we may easily calculate the eigenvalues at any $\mathbf k$ (in reduced coordinates $\mathbf k\in]-0.5:0.5]$).
Can you write Python code for it?


I am working on the file "TB_01/run.ipynb".
The first task is:
Instead of manually defining the graphene system with associated atomic coordinates and lattice vectors we use the build-in sisl capability of defining the graphene structure with a default atomic distance of $d = 1.42\,A$.
Can you write Python code for it?

graphene = sisl.geom.graphene()

print(graphene)


I am working on the file "TB_01/run.ipynb".
The first task is:
Now H is a Hamiltonian object. It works equivalently to a matrix and one may assign elements and extract elements as though it were a matrix, we will return to the intricate details of the Hamiltonian object later.
There are only 3 more things that specifies the Hamiltonian.
1. non-zero is the current number of non zero elements specified in the Hamiltonian
2. orthogonal specifies whether an overlap matrix is associated with the Hamiltonian, i.e. False for a non-orthogonal basis-set.
3. The number of spin-components are specified via the Spin class which in this case is an unpolarized system. I.e. only one spin-component.
4. The Geometry class is repeated because the Hamiltonian is associated with the Geometry passed when instantiating the Hamiltonian (sisl.Hamiltonian(graphene)).
You are now ready to add matrix elements to the Hamilton.
Explicilty specifying the matrix elements
In this small sub-section we will specify the required matrix elements manually. You should already now know that we only plan on adding nearest neighbour interactions.
We will proceed with an on-site of $0\,\mathrm{eV}$ and a coupling element of $t=-2.7\,\mathrm{eV}$.
First we set the on-site elements:
Can you write Python code for it?


I am working on the file "TB_01/run.ipynb".
The first task is:
Some basic information of the geometry is shown above:

It contains 2 atoms, na is number of atoms
It contains 2 orbitals, no is number of orbitals
Then there is a list of unique atoms associated. In this case there is 1 unqiue specie, species: 1 while the subsequent lines specifies the number of times it is occuring in the Geometry.
nsc specifies the number of supercells which is a practical way of dealing with periodic structures.
In this case we have periodicity along the first and second lattice vector. And there are 3 images along each of these directions. The primary, and one at $+$ and $-$. It should be apparent that the values in nsc are always uneven numbers. Note that for molecules it is always [1, 1, 1].
The last item, maxR is the maximal orbital range in the geometry. In this case it is equal to maxR as listed for the atomic Carbon specie. However, in case there are more than one specie it will be the maximum for the individual species.

Now we are ready to proceed with creating the Hamiltonian for the graphene lattice. But before doing so you should now consider which type of system we are going towards? Are we going to implement a nearest, next nearest or third nearest neighbour interaction?
HINT: this should be apparent from the maxR variable above.
Can you write Python code for it?


I am working on the file "TB_01/run.ipynb".
The first task is:
This will only couple the first and second atom in the primary unit-cell. But we also require couplings from the primary unit-cell to the neighbouring supercells. Remember that nsc = [3, 3, 1].
In this case we know that the first atom couples to the (-1, 0) and (0, -1), while the second atom couples to (1, 0) and (0, 1).
The coupling elements may then be specified via:
Can you write Python code for it?


I am working on the file "TB_01/run.ipynb".
The first task is:
We find 8 non-zero elements, as there should be. Remark, even though we set the on-site terms to $0$, they are interpreted as non-zero elements due to explicitly setting them.
After having setup the Hamilton, we may easily calculate the eigenvalues at any $\mathbf k$ (in reduced coordinates $\mathbf k\in]-0.5:0.5]$).
Can you write Python code for it?


I am working on the file "TB_01/run.ipynb".
The first task is:
We may also create a bandstructure of the Hamiltonian.
First we define the path we wish to take in the Brillouin zone. Start at $\Gamma$, continue to $K$, then $M$ and finish where we started.
We will calculate the band-structure with a total of 301 points and we explicitly name the special points.
Can you write Python code for it?


I am working on the file "TB_02/run.ipynb".
The first task is:
2. Creating the device, Hamiltonian $\to$ Hamiltonian
The Geometry.tile function is an explicit method to create bigger lattices from a smaller reference latice. Howewer, the tile routine is also available to the Hamiltonian object. Not only is it much easier to use, it also presents these advantages:

It guarentees that the matrix elements are the same as the reference Hamiltonian, i.e. you need not specify the parameters to construct twice,
It is much faster when creating $>500,000$ samples from smaller reference systems,
It also requires less code which increases readability and is less prone to errors.
Can you write Python code for it?

H_device = H.tile(3, axis=1)
print(H_device)


I am working on the file "TB_02/run.ipynb".
The first task is:
After calculating the transport properties of the transport problem you may also use sisl to interact with the TBtrans output (in the *.TBT.nc file):
Can you write Python code for it?

plt.plot(tbt.E, tbt.transmission(), label='k-averaged');
plt.plot(tbt.E, tbt.transmission(kavg=tbt.kindex([0, 0, 0])), label=r'$\Gamma$');
plt.xlabel('Energy [eV]'); plt.ylabel('Transmission'); plt.ylim([0, None]); plt.legend();


I am working on the file "TB_02/run.ipynb".
The first task is:
First run of TBtrans
You should first run tbtrans like this (the RUN.fdf file is already prepared with enough input options for a successfull run):
tbtrans RUN.fdf

After TBtrans is complete a number of files will be present:

siesta.TBT.nc
    The main data-file of TBtrans, this contains all calculated quantities, and everything that can be orbital resolved is orbital resolved, such as density of states.

siesta.TBT.CC
    The energy points at which TBtrans has calculated physical quantities.

siesta.TBT.KP
    Used $k$-points and their corresponding weights for integrating the Brillouin zone.

siesta.TBT.TRANS_Left-Right
    The $k$-resolved transmission from Left to the Right electrode. This is a consecutive list of transmissions for all energy points. Each $k$-point transmission is separated with a description of the $k$-point and its weight.

siesta.TBT.AVTRANS_Left-Right
    The $k$-averaged transmission from Left to the Right electrode. This is the $k$-averaged equivalent of siesta.TBT.TRANS_Left-Right.
Can you write Python code for it?


I am working on the file "TB_02/run.ipynb".
The first task is:
For more information you may execute the following lines to view the :
help(Geometry.tile)
help(Hamiltonian.tile)

Now we have created the device electronic structure. The final step is to store it in a TBtrans readable format:
Can you write Python code for it?


I am working on the file "TB_02/run.ipynb".
The first task is:
Note that the above call of the graphene lattice is different from TB 1. In this example we will create an orthogonal graphene lattice, i.e. the lattice vectors are orthogonal to each other, unlike the minimal graphene lattice.
The minimal orthogonal graphene lattice consists of 4 Carbon atoms.
Assert that we have 16 non zero elements:
Can you write Python code for it?


I am working on the file "TB_02/run.ipynb".
The first task is:
Now a file ELEC.nc file exists in the folder and it contains all the information (and more) that TBtrans requires to construct the self-energies for the electrode.
All that is required is now the device region.
An important aspect of any transport setup is that the electrodes must not have matrix elements crossing the device region. I.e. there must not be matrix elements between any of the electrodes. This restriction is easily accommodated in tight-binding setups, but for DFT systems it is less transparent.
In this tight-binding setup it simlpy means a repetition of the electrode 3 times; 1) left electrode, 2) scattering region, 3) right electrode.
1. Creating the device, Geometry; Hamiltonian; Hamiltonian.construct
Here we tile the orthogonal graphene lattice 3 times along the second lattice vector (Python is 0-based) and subsequently construct it using the same parameters.
This method of specifying all matrix elements is the most usable and easy scheme that is available in sisl.
Can you write Python code for it?

device = graphene.tile(3, axis=1)
H_device = sisl.Hamiltonian(device)
H_device.construct([[0.1, 1.43], [0, -2.7]])
print(H_device)


I am working on the file "TB_02/run.ipynb".
The first task is:
Sometimes it may be convenient to plot the entries of the matrix to assert the symmetry and structure. The second line asserts that it is indeed a Hermitian matrix:
Can you write Python code for it?

plt.spy(H_device.Hk());
print('Hermitian deviation: ',np.amax(np.abs(H.Hk() - H.Hk().T.conj())))


I am working on the file "TB_02/run.ipynb".
The first task is:
This example will setup the required electronic structures for usage in TBtrans.
You will also learn the importance of perform $k$-point convergence tests for systems using TBtrans.
We will continue with the graphene nearest neighbour tight-binding model and perform simple transport calculations using TBtrans.
Our example will again concentrate on graphene:
Can you write Python code for it?

graphene = sisl.geom.graphene(orthogonal=True)
H = sisl.Hamiltonian(graphene)
H.construct([[0.1, 1.43], [0., -2.7]])


I am working on the file "TB_03/run.ipynb".
The first task is:
After calculating the transport properties of the transport problem you may also use sisl to interact with the TBtrans output (in the *.TBT.nc file). Please repeat the same convergence tests you performed in example 02.
What are the required k-point sampling compared to 02 for a similar transmission function ?
Can you write Python code for it?

plt.plot(tbt.E, tbt.transmission(), label='k-averaged');
plt.plot(tbt.E, tbt.transmission(kavg=tbt.kindex([0, 0, 0])), label=r'$\Gamma$');
plt.xlabel('Energy [eV]'); plt.ylabel('Transmission'); plt.ylim([0, None]) ; plt.legend();


I am working on the file "TB_03/run.ipynb".
The first task is:
For more information you may execute the following lines to view the documentation:
help(Geometry.tile)
help(Hamiltonian.tile)

Now we have created the device electronic structure. The final step is to store it in a TBtrans readable format:
Can you write Python code for it?


I am working on the file "TB_03/run.ipynb".
The first task is:
Note that the above call of the graphene lattice is different from TB 2, and similar to TB 1. In this example we will create a non-orthogonal graphene lattice, i.e. the lattice vectors are the minimal lattice vectors of graphene.
The minimal graphene lattice consists of 2 Carbon atoms.
We tile the Geometry to make it slightly bigger.
You are encouraged to draw the graphene lattice vectors, and draw an arrow in the direction of the transport (along the 2nd lattice vector). Note that one can calculate transport along non-orthogonal directions (also in TranSiesta).
Assert that we have 16 non zero elements:
Can you write Python code for it?


I am working on the file "TB_03/run.ipynb".
The first task is:
The Hamiltonian we have thus far created will be our electrode. Lets write it to a TBtrans readable file:
Can you write Python code for it?


I am working on the file "TB_04/run.ipynb".
The first task is:
Can you from the above three quantities determine whether there are any localized states in the examined system?
HINT: What is the sum of the spectral density of states ($\mathbf A_i$) compared to the Green function ($\mathbf G$) density of states?
Examining DOS on individual atoms
The total DOS is a measure of the DOS spread out in the entire atomic region. However, TBtrans calculates, and stores all DOS related quantities in orbital resolution. I.e. we are able to post-process the DOS and examine the atom and/or orbital resolved DOS.
To do this the .DOS and .ADOS routines have two important keywords, 1) atom and 2) orbital which may be used to extract a subset of the DOS quantities. For details on extracting these subset quantities please read the documentation by executing the following line:
help(tbt.DOS)

The following code will extract the DOS only on the atoms in the hole edge.
Can you write Python code for it?

DOS_edge = [tbt.DOS(atoms=edge), tbt.ADOS(0, atoms=edge), tbt.ADOS(1, atoms=edge)]
plt.plot(tbt.E, DOS_edge[0], label='G');
plt.plot(tbt.E, DOS_edge[1], label=r'$A_L$');
plt.plot(tbt.E, DOS_edge[2], label=r'$A_R$');
plt.ylim([0, None]); plt.ylabel('DOS on edge atoms [1/eV]'); plt.xlabel('Energy [eV]'); plt.legend();


I am working on the file "TB_04/run.ipynb".
The first task is:
DOS depending on the distance from the hole
We can further analyze the DOS evolution for atoms farther and farther from the hole.
The following DOS analysis will extract DOS (from $\mathbf G$) for the edge atoms, then for the nearest neighbours to the edge atoms, and for the next-nearest neighbours to the edge atoms.
Try and extend the plot to contain the DOS of the next-next-nearest neighbours to the edge atoms.
Can you write Python code for it?

# Get nearest neighbours to the edge atoms
n_edge = Hdev.edges(edge, exclude=edge)
# Get next-nearest neighbours to the edge atoms
nn_edge = Hdev.edges(n_edge, exclude=np.hstack((edge, n_edge)))
# Try and create the next-next-nearest neighbours to the edge atoms and add it to the plot
plt.plot(tbt.E, tbt.DOS(atoms=edge, norm='atom'), label='edge: G');
plt.plot(tbt.E, tbt.DOS(atoms=n_edge, norm='atom'), label='n-edge: G');
plt.plot(tbt.E, tbt.DOS(atoms=nn_edge, norm='atom'), label='nn-edge: G');
plt.ylim([0, None]); plt.ylabel('DOS [1/eV/atom]'); plt.xlabel('Energy [eV]'); plt.legend();


I am working on the file "TB_04/run.ipynb".
The first task is:
Exercises
Please carefully go through the RUN.fdf file before running TBtrans, determine what each flag means and what it tells TBtrans to output.
Now run TBtrans:
tbtrans RUN.fdf

In addition to the previous example, many more files are now being created (for all files the siesta.TBT.AV* files are the $k$-averaged equivalents. You should, while reading the below list, also be able to specify which of the fdf-flags that are responsible for the creation of which files.

siesta.TBT.ADOS_*
   The $k$-resolved spectral density of states injected from the named electrode
siesta.TBT.BDOS_*
   The $k$-resolved bulk Green function density of states for the named electrode
siesta.TBT.BTRANS_*
   The $k$-resolved bulk transmission through the named electrode
siesta.TBT.DOS
   Green function density of states
siesta.TBT.TEIG_<1>-<2>
   Transmission eigenvalues for electrons injected from <1> and emitted in <2>.

This exercise mainly shows a variety of methods useful for extracting data from the *.TBT.nc files in a simple and consistent manner. You are encouraged to play with the routines, because the next example forces you to utilize them.
Can you write Python code for it?


I am working on the file "TB_04/run.ipynb".
The first task is:
In this example we will begin to cover some of the extraction utilities in sisl allowing one to really go in-depth on analysis of calculations.
We will begin by creating a large graphene flake, then subsequently a hole will be created by removing a circular shape.
Subsequent to the calculations you are encouraged to explore the sisl toolbox for ways to extract important information regarding your system.
Can you write Python code for it?

graphene = sisl.geom.graphene(orthogonal=True)

elec = graphene.tile(25, axis=0)
H = sisl.Hamiltonian(elec)
H.construct(([0.1, 1.43], [0., -2.7]))
H.write('ELEC.nc')

device = elec.tile(15, axis=1)
device = device.remove(
    device.close(
        device.center(what='cell'), R=10.)
)


I am working on the file "TB_04/run.ipynb".
The first task is:
We will also (for physical reasons) remove all dangling bonds, and secondly we will create a list of atomic indices which corresponds to the atoms at the edge of the hole.
Can you write Python code for it?

dangling = [ia for ia in device.close(device.center(what='cell'), R=14.)
                if len(device.close(ia, R=1.43)) < 3]
device = device.remove(dangling)
edge = []
for ia in device.close(device.center(what='cell'), R=14.):
    if len(device.close(ia, R=1.43)) < 4:
        edge.append(ia)
edge = np.array(edge)

# Pretty-print the list of atoms (for use in sdata)
# Note we add 1 to get fortran indices
print(sisl.utils.list2str(edge + 1))

Hdev = sisl.Hamiltonian(device)
Hdev.construct(([0.1, 1.43], [0, -2.7]))
Hdev.geometry.write('device.xyz')
Hdev.write('DEVICE.nc')


I am working on the file "TB_05/run.ipynb".
The first task is:
In this example you will learn how to make use of the periodicity of the electrodes.
As seen in TB 4 the transmission calculation takes a considerable amount of time. In this example we will redo the same calculation, but speed it up (no approximations made).
A large computational effort is made on calculating the self-energies which basically is inverting, multiplying and adding matrices, roughly 10-20 times per $k$-point, per energy point, per electrode.
For systems with large electrodes compared to the full device, this becomes more demanding than calculating the Green function for the system.
When there is periodicity in electrodes along the transverse semi-infinite direction (not along the transport direction) one can utilize Bloch's theorem to reduce the computational cost of calculating the self-energy.

In ANY calculation if you have periodicity, please USE it.

In this example you should scour the tbtrans manual on how to enable Bloch's
theorem, and once enabled it should be roughly 3 - 4 times as fast, something that is non-negligeble for large systems.
Can you write Python code for it?


I am working on the file "TB_05/run.ipynb".
The first task is:
Note the below lines are differing from the same lines in TB 4, i.e. we save the electrode electronic structure without extending it 25 times.
Can you write Python code for it?

H_elec = sisl.Hamiltonian(graphene)
H_elec.construct(([0.1, 1.43], [0., -2.7]))
H_elec.write('ELEC.nc')


I am working on the file "TB_05/run.ipynb".
The first task is:
See TB 2 for details on why we choose repeat/tile on the Hamiltonian object and not on the geometry, prior to construction.
Can you write Python code for it?

H = H_elec.repeat(25, axis=0).tile(15, axis=1)
H = H.remove(
    H.geometry.close(
        H.geometry.center(what='cell'), R=10.)
)

dangling = [ia for ia in H.geometry.close(H.geometry.center(what='cell'), R=14.)
                if len(H.edges(ia)) < 3]
H = H.remove(dangling)
edge = [ia for ia in H.geometry.close(H.geometry.center(what='cell'), R=14.)
         if len(H.edges(ia)) < 4]
edge = np.array(edge)

# Pretty-print the list of atoms
print(sisl.utils.list2str(edge + 1))

H.geometry.write('device.xyz')
H.write('DEVICE.nc')


I am working on the file "TB_06/run.ipynb".
The first task is:
Bulk density of states:
Can you write Python code for it?

Eplot(..., label=r'$BDOS_1$');
Eplot(..., label=r'$BDOS_2$');
plt.ylabel('DOS [1/eV/N]'); plt.xlabel('Energy [eV]'); plt.ylim([0, None]); plt.legend();


I am working on the file "TB_06/run.ipynb".
The first task is:
Considering symmetries, try to figure out which transmissions ($T_{ij}$) are unique?
Plot the bulk DOS for the 2 differing electrodes.
Plot the spectral DOS injected by all 4 electrodes.
Can you write Python code for it?

# Helper routines, this makes BDOS(...) == tbt.BDOS(..., norm='atom')
BDOS = partial(tbt.BDOS, norm='atom')
ADOS = partial(tbt.ADOS, norm='atom')


I am working on the file "TB_06/run.ipynb".
The first task is:
For 2D structures one can easily plot the DOS per atom via a scatter plot in matplotlib, here is the skeleton code for that, you should select an energy point and figure out how to extract the atom resolved DOS (you will need to look-up the documentation for the ADOS method to figure out which flag to use.
Can you write Python code for it?

Eidx = tbt.Eindex(...)
ADOS = [tbt.ADOS(i, ....) for i in range(4)]
f, axs = plt.subplots(2, 2, figsize=(10, 10))
a_xy = tbt.geometry.xyz[tbt.a_dev, :2]
for i in range(4):
    A = ADOS[i]
    A *= 100 / A.max() # normalize to maximum 100 (simply for plotting)
    axs[i // 2][i % 2].scatter(a_xy[:, 0], a_xy[:, 1], A, c="bgrk"[i], alpha=.5);
plt.xlabel('x [Ang]'); plt.ylabel('y [Ang]'); plt.axis('equal');


I am working on the file "TB_06/run.ipynb".
The first task is:
Make easy function calls for plotting energy resolved quantites:
Can you write Python code for it?

E = tbt.E
Eplot = partial(plt.plot, E)

# Make a shorthand version for the function (simplifies the below line)
T = tbt.transmission
t12, t13, t14, t23, t24, t34 = T(0, 1), T(0, 2), T(0, 3), T(1, 2), T(1, 3), T(2, 3)
Eplot(t12, label=r'$T_{12}$'); Eplot(t13, label=r'$T_{13}$'); Eplot(t14, label=r'$T_{14}$');
Eplot(t23, label=r'$T_{23}$'); Eplot(t24, label=r'$T_{24}$');
Eplot(t34, label=r'$T_{34}$');
plt.ylabel('Transmission'); plt.xlabel('Energy [eV]'); plt.ylim([0, None]); plt.legend();


I am working on the file "TB_06/run.ipynb".
The first task is:
Since this system has 4 electrodes we need to tell tbtrans where the 4 electrodes are in the device. The following lines prints out the fdf-lines that are appropriate for each of the electrodes (RUN.fdf is already filled correctly):
Can you write Python code for it?

print('elec-Y-1:   semi-inf -A2: {}'.format(1))
print('elec-Y-2:   semi-inf +A2: end {}'.format(len(dev_y)))
print('elec-X-1:   semi-inf -A1: {}'.format(len(dev_y) + 1))
print('elec-X-2:   semi-inf +A1: end {}'.format(-1))

H = sisl.Hamiltonian(device)
H.construct([R, hop])
H.write('DEVICE.nc')


I am working on the file "TB_06/run.ipynb".
The first task is:
Spectral density of states for all electrodes:
- As a final exercise you can explore the details of the density of states for single atoms. Take for instance atom 205 (204 in Python index) which is in both GNRs at the crossing.
Feel free to play around with different atoms, subset of atoms (pass a list) etc.
Can you write Python code for it?

Eplot(..., label=r'$ADOS_1$');
...
plt.ylabel('DOS [1/eV/N]'); plt.xlabel('Energy [eV]'); plt.ylim([0, None]); plt.legend();


I am working on the file "TB_06/run.ipynb".
The first task is:
Subsequently we create the electronic structure.
Can you write Python code for it?

H_y = sisl.Hamiltonian(elec_y)
H_y.construct((R, hop))
H_y.write('ELEC_Y.nc')
H_x = sisl.Hamiltonian(elec_x)
H_x.construct((R, hop))
H_x.write('ELEC_X.nc')


I am working on the file "TB_06/run.ipynb".
The first task is:
TBtrans is capable of calculating transport in $N\ge 1$ electrode systems. In this example we will explore a 4-terminal graphene GNR cross-bar (one zGNR, the other aGNR) system.
Can you write Python code for it?

graphene = sisl.geom.graphene(orthogonal=True)

R = [0.1, 1.43]
hop = [0., -2.7]


I am working on the file "TB_07/run.ipynb".
The first task is:
Exercises

Calculate all physical quantities for all different applied magnetic fields.
   Before running the calculations, search the manual on how to save the self-energies (HINT out-of-core). By default, TBtrans calculates the self-energies as they are needed. However, if one has the same electrodes, same $k$-grid and same $E$-points for several different runs (as in this case) one can with benefit calculate the self-energies once, and then reuse them in subsequent calculations.

To ease the calculation of all magnetic fields a script (run.sh) is located in this directory which will loop over the different fields. Carefully read it to infer which option specifies the $\delta \mathbf H$ term.
Since this example has 14 different setups, each with 51 energy points, it will take some time. Around 30 seconds for the first (includes self-energy calculation), and around 10 seconds for all subsequent setups. So be patient. :)
- Secondly, read in all output into the workbook in a list.
- TIME (HARD):
    Choose an energy-point for $B=0$ and $B>0$ such that you have a finite transmission ($T>0$). Next, extract the bond-currents and calculate the transmission using the law of current conservation (Kirchhoffs 1st law: $T_{\mathrm{in}} \equiv T_{\mathrm{out}}$). HINT: select a line of atoms and calculate the bond-currents flowing between this line and its neighbouring line of atoms. Assert that the transmission is the same using both methods.
Can you write Python code for it?

# Create short-hand function
gs = sisl.get_sile
# No magnetic field
tbt0 = gs('siesta.TBT.nc')
# All magnetic fields in increasing order
tbts = [gs('M_{}/siesta.TBT.nc'.format(rec_phi)) for rec_phi in rec_phis]


I am working on the file "TB_07/run.ipynb".
The first task is:
Make a contour plot of the transmission vs. $\Phi$ and $E$
Can you write Python code for it?

T = np.stack([tbt.transmission() for tbt in tbts])
T[T < 1e-9] = 0 # small numbers are difficult to interpolate (they tend to blow up), so remove them
plt.contourf(rec_phis, E, T.T, 20); plt.colorbar(label='Transmission');
plt.ylabel('Energy [eV]'); plt.xlabel(r'$1/\Phi$');


I am working on the file "TB_07/run.ipynb".
The first task is:
Now we have the coupling dependent phase factor, $\delta x^-\cdot\delta y^+$, and all we need to calculate is $\delta\mathbf H$ that transforms $\mathbf H(0) \to \mathbf H(\Phi)$.
This is easily done as the Hamiltonian allows basic element wise operations, i.e. +, -, *, / and ** (the power function).
Your task is to insert the correct mathematical equation below, such that dH contains $\delta \mathbf H$ for $\mathbf H(\Phi) = \mathbf H(0) + \delta\mathbf H$.
To help you I have inserted the exponential function. To finalize the equation, you need three terms: H, xy and phi (note we use the reciprocal values as loop variables for ease):
<!-- dH = np.exp(0.5j * phi * xy) * H - H-->
Can you write Python code for it?

rec_phis = np.arange(1, 51, 4)
for i, rec_phi in enumerate(tqdm(rec_phis, unit="M")):
    phi = 1 / rec_phi
    # Calculate H(Phi)
    # TODO: insert the correct mathematical equation below to calculate the correct phase
    #dH = np.exp ...
    with sisl.get_sile('M_{}.dH.nc'.format(rec_phi), mode='w') as fh:
        fh.write_delta(dH)


I am working on the file "TB_07/run.ipynb".
The first task is:
One of the key ideas behind sisl was the interaction of a DFT Hamiltonian and the user.
In this example we will highlight a unique implementation in TBtrans which enables any kind of user intervention.
The idea is a transformation of the Green function calculation from:
\begin{equation}
   \mathbf G^{-1}(E) = \mathbf S (E + i\eta) - \mathbf H - \sum_i\boldsymbol\Sigma_i
\end{equation}
to
\begin{equation}
   \mathbf G^{-1}(E) = \mathbf S (E + i\eta) - \mathbf H - \delta\mathbf H - \sum_i\boldsymbol\Sigma_i - \delta\boldsymbol \Sigma
\end{equation}
where $\delta\mathbf H$ and $\delta\boldsymbol\Sigma$ can be of any type, i.e. complex and/or real.
The only (important!) difference between $\delta\mathbf H$ and $\delta\boldsymbol \Sigma$ is that the former enters the calculation of bond-currents, while the latter does not.
Since TBtrans by it-self does not allow complex Hamiltonians the above is a way to leviate this restriction. One feature this may be used for is by applying magnetic fields.
In the following we will use Peierls substitution on a square tight-binding model:
\begin{equation}
   \mathbf H(\Phi) = \mathbf H(0)e^{i \Phi \delta x^- \cdot \delta y^+ / 2},
\end{equation}
where $\Phi$ is the magnetic flux (in proper units), $\delta x^- = x_j - x_i$ and $\delta y^+=y_j + y_i$, with $i$ and $j$ being atomic indices. If you are interested in this substitution, please search litterature.

First create a square lattice and define the on-site and nearest neighbour couplings
Can you write Python code for it?

square = sisl.Geometry([0,0,0], sisl.Atom(1, R=1.0), sc=sisl.SuperCell(1, nsc=[3, 3, 1]))
on, nn = 4, -1

H_minimal = sisl.Hamiltonian(square)
H_minimal.construct([[0.1, 1.1], [on, nn]])
H_elec = H_minimal.tile(100, 1).tile(2, 0)
H_elec.set_nsc([3, 1, 1])
H_elec.write('ELEC.nc')

H = H_elec.tile(50, 0)

# Make a constriction
geom = H.geometry.translate( -H.geometry.center(what='xyz') )
# This constriction is based on an example in the kwant project (called qhe). We however make a slight modification.
# Remove some atoms, this will create a constriction of 100 - 40 * 2 = 20 Ang with a Gaussian edge profile
remove = (np.abs(geom.xyz[:, 1]) > 50 - 37.5 * np.exp( -(geom.xyz[:, 0] / 12) **2 )).nonzero()[0]
# To reduce computations we find the atoms in the constriction such that we can
# limit the calculation region.
device = (np.abs(geom.remove(remove).xyz[:, 0]) < .6).nonzero()[0]
geom.remove(remove).write('test.xyz')

# Pretty print a range of atoms that is the smallest device region
# note this is fortran indices
print(sisl.utils.list2str(device + 1))

H = H.remove(remove)
H.finalize()
H.write('DEVICE.nc')


I am working on the file "TB_08/run.ipynb".
The first task is:
Before we can run calculations we need to create the real space self-energy for the graphene flake in sisl.
Since the algorithm is not implemented in TBtrans (nor TranSiesta) it needs to be done here.
This is somewhat complicated since the files requires a specific order. For ease this tutorial implements it for you.
Can you write Python code for it?

# A real space transport calculation ONLY needs the Gamma-point
gamma = sisl.MonkhorstPack(H_elec, [1] * 3)
# Energy contour
dE = 0.04
E = np.arange(-2, 2 + dE / 2, dE)
sisl.io.tableSile("contour.E", 'w').write_data(E, np.zeros(E.size) + dE)

# Now create the file (should take around 3-4 minutes)
eta = 0.001 * 1j
with sisl.io.tbtgfSileTBtrans("GRAPHENE.TBTGF") as f:
    f.write_header(gamma, E + eta)
    for ispin, new_k, k, e in tqdm(f, unit="rsSE"):
        if new_k:
            f.write_hamiltonian(H_elec.Hk(format='array', dtype=np.complex128))
        SeHSE = RSSE.self_energy(e + eta, bulk=True, coupling=True)
        f.write_self_energy(SeHSE)


I am working on the file "TB_08/run.ipynb".
The first task is:
In TBtrans and TranSiesta one is capable of performing real space transport calculations by using real space self-energies (see here).
Currently the real space self-energy calculation has to be performed in sisl since it is not implemented in TranSiesta.
A real space self-energy is a $\mathbf k$ averaged self-energy which can emulate any 2D or 3D electrode. I.e. for an STM junction a tip and a surface. In such a system the surface could be modelled using the real space self-energy to remove mirror effects of STM tips. This is important since the distance between periodic images disturbs the calculation due to long range potential effects.
The basic principle for calculating the real space self-energy is the Brillouin zone integral:
\begin{equation}
   \mathbf G_{\mathcal R}(E) = \int_{\mathrm{BZ}}\mathbf G_\mathbf k
\end{equation}
In this example we will construct an STM tip probing a graphene flake.

This example is rather complicated and is the reason why basically everything is already done for you. Please try and understand each step.
We start by creating the graphene tight-binding model.
Can you write Python code for it?

graphene = sisl.geom.graphene(orthogonal=True)

# Graphene tight-binding parameters
on, nn = 0, -2.7
H_minimal = sisl.Hamiltonian(graphene)
H_minimal.construct([[0.1, 1.44], [on, nn]])


I am working on the file "TB_08/run.ipynb".
The first task is:
Lastly, we need to add the STM tip. Here we simply add a gold atom and manually add the hoppings. Since this is tight-binding we have full control over the self-energy and potential land-scape. Therefore we don't need to extend the electrode region to screen off the tip region. In DFT systems, a properly screened region is required.
Can you write Python code for it?

STM = sisl.Geometry([0, 0, 0], atoms=sisl.Atom('Au', R=1.0001), sc=sisl.SuperCell([10, 10, 1], nsc=[1, 1, 3]))
H_STM = sisl.Hamiltonian(STM)
H_STM.construct([(0.1, 1.1), (0, -0.75)])
H_STM.write('STM.nc')

mid_xyz = H.geometry.center()
idx = H.close(mid_xyz, R=1.33)[0]
H_device = H.add(H_STM, offset=H.geometry.xyz[idx] - H_STM.geometry.xyz[0] + [0, 0, 2])
na = len(H)
idx = H_device.close(na, R=(0.1, 2.25))[1][0]
H_device[na, idx] = -0.1
H_device[idx, na] = -0.1

H_device.write('DEVICE.nc')


I am working on the file "TB_08/run.ipynb".
The first task is:
Now we can create the real space self-energy.
In TBtrans (and TranSiesta) the electrode atomic indices must be in consecutive order.
This is a little troublesome since the natural order in a device would be an order according to $x$, $y$ or $z$. To create the correct order we extract the real space coupling matrix which is where the real space self-energy would live, the self-energy is calculated using:
\begin{equation}
   \boldsymbol\Sigma^{\mathcal R} = E \mathbf S - \mathbf H - \Big[\int_{\mathrm{BZ}} \mathbf G\Big]^{-1}.
\end{equation}
Another way to calculate the self-energy would be to transfer the Green function from the infinite bulk into the region of interest:
\begin{equation}
   \boldsymbol\Sigma^{\mathcal R} = \mathbf V_{\mathcal R\infty}\mathbf G_{\infty\setminus\mathcal R}\mathbf V_{\infty\mathcal R}.
\end{equation}
From the 2nd equation it is obvious that the self-energy only lives on the boundary that $\mathbf V_{\infty\mathcal R}$ couples to. Exactly this region is extracted using real_space_coupling as below. Take some time to draw a simple 2D lattice coupling and confirm the area that the real-space self energies couples to.
In this example we also retrieve the indices for the electrode atoms, those that connect out to the infinite plane.
Can you write Python code for it?

H_elec, elec_indices = RSSE.real_space_coupling(ret_indices=True)
H_elec.write('GRAPHENE.nc')


I am working on the file "TB_08/run.ipynb".
The first task is:
Once the minimal graphene unit-cell (here orthogonal) is created we now turn to the calculation of the real space self-energy.
The construction of this object is somewhat complicated and has a set of required input options:
- object: the Hamiltonian
- semi_axes: which axes to use for the recursive self-energy
- k_axis: which axis to integrate in the Brillouin zone
- unfold: how many times the object needs to be unfolded along each lattice vector, this is an integer vector of length 3
Can you write Python code for it?

# object = H_minimal
# semi_axes = 0, x-axis uses recursive self-energy calculation
# k_axis = 1, y-axis uses a Brillouin zone integral
# unfold = (10, 10, 1), the full real-space green function is equivalent to the system
#                       H_minimal.tile(10, 0).tile(10, 1)
RSSE = sisl.RealSpaceSE(H_minimal, 0, 1, (10, 10, 1))


I am working on the file "TB_08/run.ipynb".
The first task is:
The above yields the electrode region which contains the self-energies. Since the full device region is nothing but the H_minimal tiled $10\times10$ times with an attached STM tip on top. Here we need to arange the electrode atoms first, then the final device region. The real_space_parent method returns the Hamiltonian that obeys the unfolded size. In this case $10\times10$ times larger. One should always use this method to get the correct device order of atoms since the order of tiling is determined by the semi_axes and k_axis arguments.
Can you write Python code for it?

H = RSSE.real_space_parent()
# Create the true device by re-arranging the atoms
indices = np.arange(len(H))
indices = np.delete(indices, elec_indices)
# first electrodes, then rest of device
indices = np.concatenate([elec_indices, indices])
# Now re-arange matrix
H = H.sub(indices)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
5.1.2. Independent & Target variable¶
Set independent, target & ID features
Can you write Python code for it?

target_labels = list(features_types_group["TARGET"])
target_id = ["patientID"]

features["train_indep"] = dict()
features["train_target"] = dict()
features["train_id"] = dict()
features["test_indep"] = dict()
features["test_target"] = dict()
features["test_id"] = dict()

# Independent and target features
def set_features_indep_target(df):
    df_targets = pd.DataFrame(dict(zip(target_labels, [[]] * len(target_labels))))
    for i in range(len(target_labels)):
        df_targets[target_labels[i]] = df[target_labels[i]]

    df_indep = df.drop(target_labels + target_id, axis=1)
    df_id = pd.DataFrame({target_id[0]: df[target_id[0]]})

    return df_indep, df_targets, df_id

# train & test sets
features["train_indep"], features["train_target"], features["train_id"] = set_features_indep_target(features["train"])
features["test_indep"], features["test_target"], features["test_id"] = set_features_indep_target(features["test"])

# print
print("Number of columns: ", len(features["train_indep"].columns))
print("features: {train: ", len(features["train_indep"]), ", test: ", len(features["test_indep"]), "}")


I am working on the file "TCARER_Basic.ipynb".
The first task is:
5.2. Remove - Near Zero Variance
In order to reduce sparseness and invalid features, highly stationary ones were withdrawn. The features that had constant counts less than or equal a threshold were ltered out, to exclude highly constants and near-zero variances.
The near zero variance rules are presented in below:
- Frequency ratio: The frequency of the most prevalent value over the second most frequent value to be greater than a threshold;
- Percent of unique values: The number of unique values divided by the total number of samples to be greater than the threshold
Configure: the function
- The cutoff for the percentage of distinct values out of the number of total samples (upper limit). e.g. 10 * 100 / 100

	 → thresh_unique_cut
- The cutoff for the ratio of the most common value to the second most common value (lower limit). eg. 95/5

	 → thresh_freq_cut
Can you write Python code for it?

thresh_unique_cut = 100
thresh_freq_cut = 1000

excludes = []
file_name = "Step_05_Preprocess_NZV_config"
features["train_indep"], o_summaries = preprocess.near_zero_var_df(df=features["train_indep"],
                                                             excludes=excludes,
                                                             file_name=file_name,
                                                             thresh_unique_cut=thresh_unique_cut,
                                                             thresh_freq_cut=thresh_freq_cut,
                                                             to_search=True)

file_name = "Step_05_Preprocess_NZV"
readers_writers.save_text(path=CONSTANTS.io_path, title=file_name, data=o_summaries, append=False, ext="log")

file_name = "Step_05_Preprocess_NZV_config"
features["test_indep"], o_summaries = preprocess.near_zero_var_df(df=features["test_indep"],
                                                            excludes=excludes,
                                                            file_name=file_name,
                                                            thresh_unique_cut=thresh_unique_cut,
                                                            thresh_freq_cut=thresh_freq_cut,
                                                            to_search=False)

# print
print("Number of columns: ", len(features["train_indep"].columns))
print("features: {train: ", len(features["train_indep"]), ", test: ", len(features["test_indep"]), "}")


I am working on the file "TCARER_Basic.ipynb".
The first task is:
5.4. Descriptive Statistics
Produce a descriptive stat report of 'Categorical', 'Continuous', & 'TARGET' features
Can you write Python code for it?

# columns
file_name = "Step_05_Data_ColumnNames_Train"
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name,
                         data=list(features["train_indep"].columns.values), append=False)

# Sample - Train
file_name = "Step_05_Stats_Categorical_Train"
o_stats = preprocess.stats_discrete_df(df=features["train_indep"], includes=features_types_group["CATEGORICAL"],
                                       file_name=file_name)
file_name = "Step_05_Stats_Continuous_Train"
o_stats = preprocess.stats_continuous_df(df=features["train_indep"], includes=features_types_group["CONTINUOUS"],
                                         file_name=file_name)

# Sample - Test
file_name = "Step_05_Stats_Categorical_Test"
o_stats = preprocess.stats_discrete_df(df=features["test_indep"], includes=features_types_group["CATEGORICAL"],
                                       file_name=file_name)
file_name = "Step_05_Stats_Continuous_Test"
o_stats = preprocess.stats_continuous_df(df=features["test_indep"], includes=features_types_group["CONTINUOUS"],
                                         file_name=file_name)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
5.5. Save Samples
Serialise & save the samples before any feature transformation.
<br/>This snapshot of the samples may be used for the population profiling
Can you write Python code for it?

file_name = "Step_05_Features"
readers_writers.save_serialised_compressed(path=CONSTANTS.io_path, title=file_name, objects=features)

# print
print("Number of columns: ", len(features["train_indep"].columns),
      "features: {train: ", len(features["train_indep"]), ", test: ", len(features["test_indep"]), "}")


I am working on the file "TCARER_Basic.ipynb".
The first task is:
6.2. Remove - Near Zero Variance
Optional: Remove more features with near zero variance, after the factorisation step.
Configure: the function
Can you write Python code for it?

# the cutoff for the percentage of distinct values out of the number of total samples (upper limit). e.g. 10 * 100 / 100
thresh_unique_cut = 100
# the cutoff for the ratio of the most common value to the second most common value (lower limit). eg. 95/5
thresh_freq_cut = 1000

excludes = []
file_name = "Step_06_Preprocess_NZV_config"
features["train_indep"], o_summaries = preprocess.near_zero_var_df(df=features["train_indep"],
                                                             excludes=excludes,
                                                             file_name=file_name,
                                                             thresh_unique_cut=thresh_unique_cut,
                                                             thresh_freq_cut=thresh_freq_cut,
                                                             to_search=True)

file_name = "Step_06_Preprocess_NZV"
readers_writers.save_text(path=CONSTANTS.io_path, title=file_name, data=o_summaries, append=False, ext="log")

file_name = "Step_06_Preprocess_NZV_config"
features["test_indep"], o_summaries = preprocess.near_zero_var_df(df=features["test_indep"],
                                                            excludes=excludes,
                                                            file_name=file_name,
                                                            thresh_unique_cut=thresh_unique_cut,
                                                            thresh_freq_cut=thresh_freq_cut,
                                                            to_search=False)

# print
print("Number of columns: ", len(features["train_indep"].columns))
print("features: {train: ", len(features["train_indep"]), ", test: ", len(features["test_indep"]), "}")


I am working on the file "TCARER_Basic.ipynb".
The first task is:
6.3. Remove Highly Linearly Correlated
Optional: Remove more features with highly linearly correlated, after the factorisation step.
Configure: the function
Can you write Python code for it?

# A numeric value for the pair-wise absolute correlation cutoff. e.g. 0.95
thresh_corr_cut = 0.95

excludes = []
file_name = "Step_06_Preprocess_Corr_config"
features["train_indep"], o_summaries = preprocess.high_linear_correlation_df(df=features["train_indep"],
                                                                       excludes=excludes,
                                                                       file_name=file_name,
                                                                       thresh_corr_cut=thresh_corr_cut,
                                                                       to_search=True)

file_name = "Step_06_Preprocess_Corr"
readers_writers.save_text(path=CONSTANTS.io_path, title=file_name, data=o_summaries, append=False, ext="log")

file_name = "Step_06_Preprocess_Corr_config"
features["test_indep"], o_summaries = preprocess.high_linear_correlation_df(df=features["test_indep"],
                                                                      excludes=excludes,
                                                                      file_name=file_name,
                                                                      thresh_corr_cut=thresh_corr_cut,
                                                                      to_search=False)

# print
print("Number of columns: ", len(features["train_indep"].columns))
print("features: {train: ", len(features["train_indep"]), ", test: ", len(features["test_indep"]), "}")


I am working on the file "TCARER_Basic.ipynb".
The first task is:
6.4. Descriptive Statsistics
Produce a descriptive stat report of 'Categorical', 'Continuous', & 'TARGET' features
Can you write Python code for it?

# columns
file_name = "Step_06_4_Data_ColumnNames_Train"
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name,
                         data=list(features["train_indep"].columns.values), append=False)

# Sample - Train
file_name = "Step_06_4_Stats_Categorical_Train"
o_stats = preprocess.stats_discrete_df(df=features["train_indep"], includes=features_types_group["CATEGORICAL"],
                                       file_name=file_name)
file_name = "Step_06_4_Stats_Continuous_Train"
o_stats = preprocess.stats_continuous_df(df=features["train_indep"], includes=features_types_group["CONTINUOUS"],
                                         file_name=file_name)

# Sample - Test
file_name = "Step_06_4_Stats_Categorical_Test"
o_stats = preprocess.stats_discrete_df(df=features["test_indep"], includes=features_types_group["CATEGORICAL"],
                                       file_name=file_name)
file_name = "Step_06_4_Stats_Continuous_Test"
o_stats = preprocess.stats_continuous_df(df=features["test_indep"], includes=features_types_group["CONTINUOUS"],
                                         file_name=file_name)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
6.6. Summary Statistics
Produce a descriptive stat report of 'Categorical', 'Continuous', & 'TARGET' features
Can you write Python code for it?

# Statsistics report for 'Categorical', 'Continuous', & 'TARGET' variables
# columns
file_name = "Step_06_6_Data_ColumnNames_Train"
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name,
                         data=list(features["train_indep"].columns.values), append=False)

# Sample - Train
file_name = "Step_06_6_Stats_Categorical_Train"
o_stats = preprocess.stats_discrete_df(df=features["train_indep"], includes=features_types_group["CATEGORICAL"],
                                       file_name=file_name)
file_name = "Step_06_6_Stats_Continuous_Train"
o_stats = preprocess.stats_continuous_df(df=features["train_indep"], includes=features_types_group["CONTINUOUS"],
                                         file_name=file_name)

# Sample - Test
file_name = "Step_06_6_Stats_Categorical_Test"
o_stats = preprocess.stats_discrete_df(df=features["test_indep"], includes=features_types_group["CATEGORICAL"],
                                       file_name=file_name)
file_name = "Step_06_6_Stats_Continuous_Test"
o_stats = preprocess.stats_continuous_df(df=features["test_indep"], includes=features_types_group["CONTINUOUS"],
                                         file_name=file_name)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
7.1. Define
Ranking Method: Random forest classifier (Brieman)

Define a set of classifiers with different settings, to be used in feature ranking trials.
Can you write Python code for it?

def rank_random_forest_brieman(features_indep_arg, features_target_arg, num_trials):
    num_settings = 3
    o_summaries_df = [pd.DataFrame({'Name': list(features_indep_arg.columns.values)}) for _ in range(num_trials * num_settings)]
    model_rank = [None] * (num_trials * num_settings)

    # trials
    for i in range(num_trials):
        print("Trial: " + str(i))
        # setting-1
        s_i = i
        model_rank[s_i] = feature_selection.rank_random_forest_breiman(
            features_indep_arg.values, features_target_arg.values,
            **{"n_estimators": 10, "criterion": 'gini', "max_depth": None, "min_samples_split": 2, "min_samples_leaf": 1,
            "min_weight_fraction_leaf": 0.0, "max_features": 'auto', "max_leaf_nodes": None, "bootstrap": True,
            "oob_score": False, "n_jobs": -1, "random_state": None, "verbose": 0, "warm_start": False, "class_weight": None})

        # setting-2
        s_i = num_trials + i
        model_rank[s_i] = feature_selection.rank_random_forest_breiman(
            features_indep_arg.values, features_target_arg.values,
            **{"n_estimators": 10, "criterion": 'gini', "max_depth": None, "min_samples_split": 50, "min_samples_leaf": 25,
            "min_weight_fraction_leaf": 0.0, "max_features": 'auto', "max_leaf_nodes": None, "bootstrap": True,
            "oob_score": False, "n_jobs": -1, "random_state": None, "verbose": 0, "warm_start": False, "class_weight": None})

        # setting-3
        s_i = (num_trials * 2) + i
        model_rank[s_i] = feature_selection.rank_random_forest_breiman(
            features_indep_arg.values, features_target_arg.values,
            **{"n_estimators": 10, "criterion": 'gini', "max_depth": None, "min_samples_split": 40, "min_samples_leaf": 20,
            "min_weight_fraction_leaf": 0.0, "max_features": 'auto', "max_leaf_nodes": None, "bootstrap": True,
            "oob_score": False, "n_jobs": -1, "random_state": None, "verbose": 0, "warm_start": True, "class_weight": None})

    for i in...


I am working on the file "TCARER_Basic.ipynb".
The first task is:
7.3. Summaries
Can you write Python code for it?

# combine scores
def rank_summarise (features_arg, o_summaries_df_arg):
    summaries_temp = {'Order_avg': [], 'Order_max': [],  'Order_min': [], 'Importance_avg': []}
    summary_order = []
    summary_importance = []

    for f_name in list(features_arg.columns.values):
        for i in range(len(o_summaries_df_arg)):
            summary_order.append(o_summaries_df_arg[i][o_summaries_df_arg[i]['Name'] == f_name]['Order'].values)
            summary_importance.append(o_summaries_df_arg[i][o_summaries_df_arg[i]['Name'] == f_name]['Importance'].values)

        summaries_temp['Order_avg'].append(statistics.mean(np.concatenate(summary_order)))
        summaries_temp['Order_max'].append(max(np.concatenate(summary_order)))
        summaries_temp['Order_min'].append(min(np.concatenate(summary_order)))
        summaries_temp['Importance_avg'].append(statistics.mean(np.concatenate(summary_importance)))

    summaries_df = pd.DataFrame({'Name': list(features_arg.columns.values)})
    summaries_df['Order_avg'] = summaries_temp['Order_avg']
    summaries_df['Order_max'] = summaries_temp['Order_max']
    summaries_df['Order_min'] = summaries_temp['Order_min']
    summaries_df['Importance_avg'] = summaries_temp['Importance_avg']
    summaries_df = summaries_df.sort_values(['Order_avg'], ascending = [1])
    return summaries_df

# combine scores
summaries_df = dict()

for rank_model in o_summaries_df.keys():
    summaries_df[rank_model] = dict()
    summaries_df[rank_model] = rank_summarise(features["train_indep"], o_summaries_df[rank_model])


I am working on the file "TCARER_Basic.ipynb".
The first task is:
7.4. Select Top Features
Configure: the selection method
Can you write Python code for it?

rank_model = "rfc"
file_name = "Step_07_Top_Features_" + rank_model
rank_top_features_max = 400
rank_top_features_score_min = 0.1 * (10 ^ -20)

# sort features
features_names_selected = summaries_df[rank_model]['Name'][summaries_df[rank_model]['Order_avg'] >= rank_top_features_score_min]
features_names_selected = (features_names_selected[0:rank_top_features_max]).tolist()


I am working on the file "TCARER_Basic.ipynb".
The first task is:
7.5. Summary Statistics
Produce a descriptive stat report of 'Categorical', 'Continuous', & 'TARGET' features
Can you write Python code for it?

# columns
file_name = "Step_07_Data_ColumnNames_Train"
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name,
                         data=list(features["train_indep"][features_names_selected].columns.values), append=False)

# Sample - Train
file_name = "Step_07_Stats_Categorical_Train"
o_stats = preprocess.stats_discrete_df(df=features["train_indep"][features_names_selected], includes=features_types_group["CATEGORICAL"],
                                       file_name=file_name)
file_name = "Step_07_Stats_Continuous_Train"
o_stats = preprocess.stats_continuous_df(df=features["train_indep"][features_names_selected], includes=features_types_group["CONTINUOUS"],
                                         file_name=file_name)

# Sample - Test
file_name = "Step_07_Stats_Categorical_Test"
o_stats = preprocess.stats_discrete_df(df=features["test_indep"][features_names_selected], includes=features_types_group["CATEGORICAL"],
                                       file_name=file_name)
file_name = "Step_07_Stats_Continuous_Test"
o_stats = preprocess.stats_continuous_df(df=features["test_indep"][features_names_selected], includes=features_types_group["CONTINUOUS"],
                                         file_name=file_name)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
7.6. Save Features
Can you write Python code for it?

file_name = "Step_07_Features"
readers_writers.save_serialised_compressed(path=CONSTANTS.io_path, title=file_name, objects=features)

# print
print("File size: ", os.stat(os.path.join(CONSTANTS.io_path, file_name + ".bz2")).st_size)
print("Number of columns: ", len(features["train_indep"].columns))
print("features: {train: ", len(features["train_indep"]), ", test: ", len(features["test_indep"]), "}")


I am working on the file "TCARER_Basic.ipynb".
The first task is:
8.1.3. Features
Can you write Python code for it?

sample_train = features["train_indep"][features_names_selected] # features["train_indep"][features_names_selected], features["train_indep"]
sample_train_target = features["train_target"][target_feature] # features["train_target"][target_feature]
sample_test = features["test_indep"][features_names_selected] # features["test_indep"][features_names_selected], features["test_indep"]
sample_test_target = features["test_target"][target_feature] # features["test_target"][target_feature]


I am working on the file "TCARER_Basic.ipynb".
The first task is:
8.4. Predict
Predict & report performance, using the test sample
Can you write Python code for it?

o_summaries = dict()
# predict
model = training_method.predict(sample_test, "test")

# short summary
o_summaries = training_method.predict_summaries(pd.Series(sample_test_target), "test")

# Print the main performance statistics
for k in o_summaries.keys():
    print(k,  o_summaries[k])

# Print the by risk-bands of a selection of statistics
o_summaries = training_method.predict_summaries_risk_bands(pd.Series(sample_test_target), "test", np.arange(0, 1.05, 0.05))
display(o_summaries)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
8.5. Cross-Validation
Perform k-fold cross-validation
Can you write Python code for it?

o_summaries = dict()
score = training_method.cross_validate(sample_test, sample_test_target, scoring="neg_mean_squared_error", cv=10)

# short summary
o_summaries = training_method.cross_validate_summaries()
print("Scores: ", o_summaries)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
8.6. Save
Save the training model.
Can you write Python code for it?


I am working on the file "TCARER_Basic.ipynb".
The first task is:

1.1.  Initialise General Settings
Main configuration Settings:
- Specify the full path of the configuration file

	 → config_path
- Specify the full path of the output folder

	 → io_path
- Specify the application name (the suffix of the outputs file name)

	 → app_name
- Specify the sub-model name, to locate the related feature configuration, based on the "Table_Reference_Name" column in the configuration file

	 → submodel_name
- Specify the sub-model's the file name of the input (excluding the CSV extension)

	 → submodel_input_name

External Configration Files:
- The MySQL database configuration setting & other configration metadata

	 → Inputs/CONFIGURATIONS_1.ini
- The input features' confugration file (Note: only the CSV export of the XLSX will be used by this Notebook)

	 → Inputs/config_features_path.xlsx

	 → Inputs/config_features_path.csv
Can you write Python code for it?

config_path = os.path.abspath("ConfigInputs/CONFIGURATIONS.ini")
io_path = os.path.abspath("../../tmp/TCARER/Basic_prototype")
app_name = "T-CARER"
submodel_name = "hesIp"
submodel_input_name = "tcarer_model_features_ip"

print("\n The full path of the configuration file: \n\t", config_path,
      "\n The full path of the output folder: \n\t", io_path,
      "\n The application name (the suffix of the outputs file name): \n\t", app_name,
      "\n The sub-model name, to locate the related feature configuration: \n\t", submodel_name,
      "\n The the sub-model's the file name of the input: \n\t", submodel_input_name)


I am working on the file "TCARER_Basic.ipynb".
The first task is:

3. Read Data
Read the input features from the CSV input file
Can you write Python code for it?

features_input = readers_writers.load_csv(path=CONSTANTS.io_path, title=submodel_input_name, dataframing=True)
features_input.astype(dtype=features_dtypes)

print("Number of columns: ", len(features_input.columns), "; Total records: ", len(features_input.index))


I am working on the file "TCARER_Basic.ipynb".
The first task is:

4. Filter Features
4.1. Descriptive Statsistics
Produce a descriptive stat report of 'Categorical', 'Continuous', & 'TARGET' features
Can you write Python code for it?

file_name = "Step_04_Data_ColumnNames"
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name, data=list(features_input.columns.values), append=False)
file_name = "Step_04_Stats_Categorical"
o_stats = preprocess.stats_discrete_df(df=features_input, includes=features_types_group["CATEGORICAL"],
                                       file_name=file_name)
file_name = "Step_04_Stats_Continuous"
o_stats = preprocess.stats_continuous_df(df=features_input, includes=features_types_group["CONTINUOUS"],
                                         file_name=file_name)
file_name = "Step_04_Stats_Target"
o_stats = preprocess.stats_discrete_df(df=features_input, includes=features_types_group["TARGET"],
                                       file_name=file_name)


I am working on the file "TCARER_Basic.ipynb".
The first task is:

6. Recategorise & Transform
Verify features visually
Can you write Python code for it?

display(pd.concat([features["train_id"].head(), features["train_target"].head(), features["train_indep"].head()], axis=1))
display(pd.concat([features["test_id"].head(), features["test_target"].head(), features["test_indep"].head()], axis=1))


I am working on the file "TCARER_Basic.ipynb".
The first task is:

7. Rank & Select Features
Configure: the general settings
Can you write Python code for it?

# select the target variable
target_feature = "label365" # "label30", "label365"

# number of trials
num_trials = 1

model_rank = dict()
o_summaries_df = dict()


I am working on the file "TCARER_Basic.ipynb".
The first task is:

8.1.  Initialise
8.1.1. Algorithms
Configure: the trianing algorithm
Algorithm 1: Random Forest
Can you write Python code for it?

method_name = "rfc"
kwargs = {"n_estimators": 20, "criterion": 'gini', "max_depth": None, "min_samples_split": 100,
    "min_samples_leaf": 50, "min_weight_fraction_leaf": 0.0, "max_features": 'auto',
    "max_leaf_nodes": None, "bootstrap": True, "oob_score": False, "n_jobs": -1, "random_state": None,
    "verbose": 0, "warm_start": False, "class_weight": "balanced_subsample"}


I am working on the file "TCARER_Basic.ipynb".
The first task is:

8.1.2. Other Settings
Configure: other modelling settings
Can you write Python code for it?

# select the target variable
target_feature = "label365" # "label30" , "label365"

# file name
file_name = "Step_09_Model_" + method_name + "_" + target_feature

# initialise
training_method = TrainingMethod(method_name)


I am working on the file "TCARER_Basic.ipynb".
The first task is:

2. Generate Features
Notes:
- It generates the final spell-wise & temporal features from the MySQL table(s), & converts it into CSV(s);
- It generates the CSV(s) based on the configuration file of the features (Note: only the CSV export of the XLSX will be used by this Notebook)

	 → Inputs/config_features_path.xlsx

	 → Inputs/config_features_path.csv
Can you write Python code for it?

skip = True

# settings
csv_schema = ["my_db_schema"]
csv_input_tables = ["tcarer_features"]
csv_history_tables = ["hesIp"]
csv_column_index = "localID"
csv_output_table = "tcarer_model_features_ip"
csv_query_batch_size =  100000

if skip is False:
    # generate the csv file
    variables = Variables(submodel_name,
                          CONSTANTS.io_path,
                          CONSTANTS.io_path,
                          CONSTANTS.config_features_path,
                          csv_output_table)
    variables.set(csv_schema, csv_input_tables, csv_history_tables, csv_column_index, csv_query_batch_size)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Ranking Method: Gradient Boosted Regression Trees (GBRT)
Note:: It is moderately resource intensive
Can you write Python code for it?

rank_model = "gbrt"
model_rank[rank_model] = dict()
o_summaries_df[rank_model] = dict()
model_rank[rank_model], o_summaries_df[rank_model] = rank_gbrt(
    features["train_indep"], features["train_target"][target_feature], num_trials)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Ranking Method: Gradient Boosted Regression Trees (GBRT)

Define a set of classifiers with different settings, to be used in feature ranking trials.
Can you write Python code for it?

def rank_gbrt(features_indep_arg, features_target_arg, num_trials):
    num_settings = 3
    o_summaries_df = [pd.DataFrame({'Name': list(features_indep_arg.columns.values)}) for _ in range(num_trials * num_settings)]
    model_rank = [None] * (num_trials * num_settings)

    # trials
    for i in range(num_trials):
        print("Trial: " + str(i))
        # setting-1
        s_i = i
        model_rank[s_i] = feature_selection.rank_tree_gbrt(
            features_indep_arg.values, features_target_arg.values,
            **{"loss": 'ls', "learning_rate": 0.1, "n_estimators": 100, "subsample": 1.0, "min_samples_split": 2, "min_samples_leaf": 1,
            "min_weight_fraction_leaf": 0.0, "max_depth": 10, "init": None, "random_state": None, "max_features": None, "alpha": 0.9,
            "verbose": 0, "max_leaf_nodes": None, "warm_start": False, "presort": True})

        # setting-2
        s_i = num_trials + i
        model_rank[s_i] = feature_selection.rank_tree_gbrt(
            features_indep_arg.values, features_target_arg.values,
            **{"loss": 'ls', "learning_rate": 0.1, "n_estimators": 100, "subsample": 1.0, "min_samples_split": 2, "min_samples_leaf": 1,
            "min_weight_fraction_leaf": 0.0, "max_depth": 5, "init": None, "random_state": None, "max_features": None, "alpha": 0.9,
            "verbose": 0, "max_leaf_nodes": None, "warm_start": False, "presort": True})

        # setting-3
        s_i = (num_trials * 2) + i
        model_rank[s_i] = feature_selection.rank_tree_gbrt(
            features_indep_arg.values, features_target_arg.values,
            **{"loss": 'ls', "learning_rate": 0.1, "n_estimators": 100, "subsample": 1.0, "min_samples_split": 2, "min_samples_leaf": 1,
            "min_weight_fraction_leaf": 0.0, "max_depth": 3, "init": None, "random_state": None, "max_features": None, "alpha": 0.9,
            "verbose": 0, "max_leaf_nodes": None, "warm_start": False, "presort": True})

    for i in range((num_trials * num_settings)):
       ...


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Tranformation: scale
Note:: It is highly resource intensive
Can you write Python code for it?

transform_type = "scale"
kwargs = {"with_mean": True}
method_args = dict()
excludes = list(features_types_group["CATEGORICAL"]) + list(features_types_group["DUMMIES"])

features["train_indep"], method_args = preprocess.transform_df(df=features["train_indep"], excludes=excludes,
                                                               transform_type=transform_type, threaded=False,
                                                               method_args=method_args, **kwargs)
features["test_indep"], _ = preprocess.transform_df(df=features["test_indep"], excludes=excludes,
                                                    transform_type=transform_type, threaded=False,
                                                    method_args=method_args, **kwargs)

# print("Metod arguments:", method_args)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Algorithm 2: Logistic Regression
Can you write Python code for it?

method_name = "lr"
kwargs = {"penalty": 'l1', "dual": False, "tol": 0.0001, "C": 1, "fit_intercept": True, "intercept_scaling": 1,
          "class_weight": None, "random_state": None, "solver": 'liblinear', "max_iter": 100, "multi_class": 'ovr',
          "verbose": 0, "warm_start": False, "n_jobs": -1}


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Algorithm 3: Logistic Cross-Validation
Can you write Python code for it?

method_name = "lr_cv"
kwargs = {"Cs": 10, "fit_intercept": True, "cv": None, "dual": False, "penalty": 'l2', "scoring": None,
          "solver": 'lbfgs', "tol": 0.0001, "max_iter": 10, "class_weight": None, "n_jobs": -1, "verbose": 0,
          "refit": True, "intercept_scaling": 1.0, "multi_class": "ovr", "random_state": None}


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Algorithm 4: Neural Network
Can you write Python code for it?

method_name = "nn"
kwargs = {"solver": 'lbfgs', "alpha": 1e-5, "hidden_layer_sizes": (5, 2), "random_state": 1}


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Algorithm 5: k-Nearest Neighbourhood
Can you write Python code for it?

method_name = "knc"
kwargs = {"n_neighbors": 5, "weights": 'distance', "algorithm": 'auto', "leaf_size": 30,
          "p": 2, "metric": 'minkowski', "metric_params": None, "n_jobs": -1}


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Algorithm 6: Decision Tree
Can you write Python code for it?

method_name = "dtc"
kwargs = {"criterion": 'gini', "splitter": 'best', "max_depth": None, "min_samples_split": 30,
        "min_samples_leaf": 30, "min_weight_fraction_leaf": 0.0, "max_features": None,
        "random_state": None, "max_leaf_nodes": None, "class_weight": None, "presort": False}


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Algorithm 8: Naive Bayes

Note: features must be positive
Can you write Python code for it?

method_name = "nb"
training_method = TrainingMethod(method_name)
kwargs = {"alpha": 1.0, "fit_prior": True, "class_prior": None}


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Clean-Up
Can you write Python code for it?

del features["train"]
del features["test"]
gc.collect()


del features["train_indep_temp"]
del features["test_indep_temp"]
gc.collect()


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Configure: the selected feature manually if it isnecessary!
Can you write Python code for it?

file_name = "Step_07_Top_Features_rfc_adhoc"

features_names_selected = readers_writers.load_csv(path=CONSTANTS.io_path, title=file_name, dataframing=False)[0]
features_names_selected = [f.replace("\n", "") for f in features_names_selected]
display(pd.DataFrame(features_names_selected))


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Import libraries
Can you write Python code for it?

# Import Python libraries
import logging
import os
import sys
import gc
import pandas as pd
from IPython.display import display, HTML
from collections import OrderedDict
import numpy as np
import statistics
from scipy.stats import stats

# Import local Python modules
from Configs.CONSTANTS import CONSTANTS
from Configs.Logger import Logger
from Features.Variables import Variables
from ReadersWriters.ReadersWriters import ReadersWriters
from Stats.PreProcess import PreProcess
from Stats.FeatureSelection import FeatureSelection
from Stats.TrainingMethod import TrainingMethod
from Stats.Plots import Plots

# Check the interpreter
print("\nMake sure the correct Python interpreter is used!")
print(sys.version)
print("\nMake sure sys.path of the Python interpreter is correct!")
print(os.getcwd())


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Initialise constants and some of classes
Can you write Python code for it?

# Initialise constants
CONSTANTS.set(io_path, app_name)

# Initialise other classes
readers_writers = ReadersWriters()
preprocess = PreProcess(io_path)
feature_selection = FeatureSelection()
plts = Plots()

# Set print settings
pd.set_option('display.width', 1600, 'display.max_colwidth', 800)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Manually add dummy variables to the dataframe & remove the original Categorical variables
Can you write Python code for it?

features["train_indep_temp"] = preprocess.factoring_feature_wise(features["train_indep"], categories_dic, labels_dic, dtypes_dic, threaded=False)
features["test_indep_temp"] = preprocess.factoring_feature_wise(features["test_indep"], categories_dic, labels_dic, dtypes_dic, threaded=False)

# print
print("Number of columns: ", len(features["train_indep"].columns))
print("features: {train: ", len(features["train_indep"]), ", test: ", len(features["test_indep"]), "}")


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Save
Can you write Python code for it?

# save to CSV
readers_writers.save_csv(path=CONSTANTS.io_path, title=file_name, data=features_names_selected, append=False, header=False)

# print
print("Number of columns: ", len(features["train_indep"].columns))
print("features: {train: ", len(features["train_indep"]), ", test: ", len(features["test_indep"]), "}")
print("List of sorted features, which can be modified:\n  " + CONSTANTS.io_path + file_name + "csv")


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Select categories: by order of freq., max_categories_frac, & max_categories_num

Configure: The input arguments are:
- Specify the maximum number of categories a feature can have

	 → max_categories_frac
- Specify the minimum number of categories a feature can have

	 → min_categories_num
- Specify to exclude the state '0' (zero). State zero in our features represents 'any other state', including NULL

	 → exclude_zero = False
Can you write Python code for it?

max_categories_frac = 0.90
min_categories_num = 1
exclude_zero = False # if possible remove state zero

categories_dic, labels_dic, dtypes_dic, features_types_group["DUMMIES"] = \
    factorise_settings(max_categories_frac, min_categories_num, exclude_zero)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Set
Can you write Python code for it?

features["train_indep"] = features["train_indep_temp"].copy(True)
features["test_indep"] = features["test_indep_temp"].copy(True)


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Verify features visually
Can you write Python code for it?

display(pd.concat([features["train_id"].head(), features["train_target"].head(), features["train_indep"].head()], axis=1))
display(pd.concat([features["test_id"].head(), features["test_target"].head(), features["test_indep"].head()], axis=1))


display(pd.concat([features["train_id"].head(), features["train_target"].head(), features["train_indep_temp"].head()], axis=1))
display(pd.concat([features["test_id"].head(), features["test_target"].head(), features["test_indep_temp"].head()], axis=1))


I am working on the file "TCARER_Basic.ipynb".
The first task is:
Verify the top features visually
Can you write Python code for it?

# print
print("Number of columns: ", len(features["train_indep"].columns),
    ";\nNumber of top columns: ", len(features["train_indep"][features_names_selected].columns))
print("features: {train: ", len(features["train_indep"][features_names_selected]), ", test: ", len(features["test_indep"][features_names_selected]), "}")


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
1.1.  Initialise General Settings
Can you write Python code for it?

config_path = os.path.abspath("ConfigInputs/CONFIGURATIONS.ini")
io_path = os.path.abspath("../../tmp/TCARER/Basic_prototype")
app_name = "T-CARER"
submodel_name = "hesIp"
submodel_input_name = "tcarer_model_features_ip"

print("\n The full path of the configuration file: \n\t", config_path,
      "\n The full path of the output folder: \n\t", io_path,
      "\n The application name (the suffix of the outputs file name): \n\t", app_name,
      "\n The sub-model name, to locate the related feature configuration: \n\t", submodel_name,
      "\n The the sub-model's the file name of the input: \n\t", submodel_input_name)

# Initialise the logs
if not os.path.exists(io_path):
    os.makedirs(io_path, exist_ok=True)

logger = Logger(path=io_path, app_name=app_name, ext="log")
logger = logging.getLogger(app_name)

# Initialise constants
CONSTANTS.set(io_path, app_name)

# Initialise other classes
readers_writers = ReadersWriters()
preprocess = PreProcess(CONSTANTS.io_path)
feature_selection = FeatureSelection()
plots = Plots()

# Set print settings
pd.set_option('display.width', 1600, 'display.max_colwidth', 800)


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
3.2. Define Base Features
Can you write Python code for it?

feature_columns = dict()

# Categorical base columns.
for name in features_types_group["CATEGORICAL"]:
    feature_columns[name] = tf.contrib.layers.sparse_column_with_hash_bucket(
        name, hash_bucket_size=len(features_states_values[name]), combiner="sqrtn")

# Continuous base columns.
for name in features_types_group["CONTINUOUS"]:
    if features_states_values[name] is not None:
        feature_columns[name] = tf.contrib.layers.real_valued_column(name)
        feature_columns[name] = tf.contrib.layers.bucketized_column(
            feature_columns[name], [int(i) for i in features_states_values[name]])
    else:
        feature_columns[name] = tf.contrib.layers.real_valued_column(name)


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
3.4. The Deep Model
Can you write Python code for it?

print(features_types_group["CATEGORICAL"])
print([name for name in features_types_group["CONTINUOUS"] if features_states_values[name] is not None])
print([name for name in features_types_group["CONTINUOUS"] if features_states_values[name] is None])

deep_columns = [feature_columns[name] for name in features_types_group["CONTINUOUS"]]

deep_columns = deep_columns + \
    [tf.contrib.layers.embedding_column(feature_columns["gender"], dimension=2),
     tf.contrib.layers.embedding_column(feature_columns["ethnos"], dimension=3),
    tf.contrib.layers.embedding_column(feature_columns["imd04rk"], dimension=5),
    tf.contrib.layers.embedding_column(feature_columns["ageTrigger"], dimension=5)]

for name in features_types_group["CATEGORICAL_EXTRA"]:
    deep_columns = deep_columns + \
        [tf.contrib.layers.embedding_column(feature_columns[name], dimension=3)]

print(deep_columns)


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:

4.1.  Initialise
Configure the size and batches of the Deep Neural Network
Can you write Python code for it?

train_batch_size = 2000
train_steps = 500 # 40000
train_num_epochs = None
train_randomize_input = True

test_batch_size = 2000
test_steps = 500 # 300
test_num_epochs = None
test_randomize_input = False

monitor_batch_size = 2000
monitor_steps = 200
monitor_num_epochs = None
monitor_randomize_input = False

dnn_hidden_units = [24000, 12000, 6000] # [20000, 16000, 10000, 8000, 7000, 6000, 5000, 4000] # [28000, 14000, 7000]  # [24000, 12000, 6000]


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:

4.2.  Fit
Train the Deep Neural Network
Can you write Python code for it?

train_randomize_input

model_dnn.fit(input_fn=lambda: input_fn(train_file_names_full, train_batch_size,
                                        train_num_epochs, train_randomize_input),
              steps=train_steps) # , monitors=[validation_mointor]


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:

4.3.  Predict - Train Sample
Test the Deep Neural Network, using the train sample
Can you write Python code for it?

results = model_dnn.evaluate(input_fn=lambda: input_fn(train_file_names_full, test_batch_size,
                                                       test_num_epochs, test_randomize_input),
                             steps=test_steps)
for key in sorted(results):
    print("%s: %s" % (key, results[key]))


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:

4.4.  Predict - Test Sample
Test the Deep Neural Network, using the test sample
Can you write Python code for it?

results = model_dnn.evaluate(input_fn=lambda: input_fn(test_file_names_full, test_batch_size,
                                                       test_num_epochs, test_randomize_input),
                             steps=test_steps)
for key in sorted(results):
    print("%s: %s" % (key, results[key]))


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:

4.5.  Save
Save the output summaries, including the predicted probabilities
4.5.1. Stats
Can you write Python code for it?

def generator_to_list(generator, max_size):
    j = 0
    temp = [None] * max_size

    for value in generator:
        temp[j] = value
        j += 1
        if j >= max_size:
            break

    return temp

summaries["train"]["predict_proba"] = \
    generator_to_list(summaries["train"]["predict_proba"], test_batch_size  * test_steps)
summaries["test"]["predict_proba"] = \
    generator_to_list(summaries["test"]["predict_proba"], test_batch_size  * test_steps)

file_name = "model_tensorflow_summaries_" + target_feature
readers_writers.save_serialised_compressed(path=CONSTANTS.io_path, title=file_name, objects=summaries)


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:

3. Set TensorFlow Settings
Configure: the Deep Neural Network nodes
3.1. Prepare Features
Update features by type
Can you write Python code for it?

# update features
names = [i for i in features_types_group["CATEGORICAL"]]
for name in names:
    if name not in features_names_selected_raw :
        features_types_group["CATEGORICAL"].remove(name)

names = [i for i in features_types_group["CONTINUOUS"]]
for name in names:
    if name not in features_names_selected_raw :
        features_types_group["CONTINUOUS"].remove(name)

print("Categorical Features: ", features_types_group["CATEGORICAL"])
print("Continuous Features: ", features_types_group["CONTINUOUS"])


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:

 Prerequisites:

Run this notebook after the feature selection (Section 7) of the TCARER_Basic Notebook!
Make sure the following files are present in the input folder:
Step_05_Features.bz2
Step_07_Top_Features_...

1.3. Load the Top Features
Configure: the selected features
Load the top features
Can you write Python code for it?

file_name = "Step_07_Top_Features_rfc_adhoc"

features_names_selected = readers_writers.load_csv(path=CONSTANTS.io_path, title=file_name, dataframing=False)[0]
features_names_selected = [f.replace("\n", "") for f in features_names_selected]
display(pd.DataFrame(features_names_selected))


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Clean-up
Can you write Python code for it?


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Additional variables to convert to discrete
Can you write Python code for it?

names = [i for i in features_types_group["CONTINUOUS"]]
features_types_group["CATEGORICAL_EXTRA"] = list()

# convet gapDay_..., & epidur_... variables
# states = [0, 3, 7, 14, 30, 60]
for name in names:
    if name[0:7] == "gapDays_" or name[0:7] == "epidur_":
        features_types_group["CONTINUOUS"].remove(name)
        features_types_group["CATEGORICAL"].add(name)
        features_types_group["CATEGORICAL_EXTRA"].append(name)
        features_states_values[name] = [0, 3, 7, 14, 30, 60]


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Combining Wide and Deep Models into One
Can you write Python code for it?

config = tf.ConfigProto(allow_soft_placement=True)

model_dnn = tf.contrib.learn.DNNLinearCombinedClassifier(
                model_dir=model_dir,
                linear_feature_columns=wide_columns,
                config=None, # tf.contrib.learn.RunConfig(save_checkpoints_secs=600)),
                dnn_feature_columns=deep_columns,
                dnn_hidden_units=dnn_hidden_units,
                dnn_optimizer=None, # tf.train.AdagradOptimizer(...)
                linear_optimizer=None, # tf.train.FtrlOptimizer(...)
                dnn_activation_fn=tf.nn.relu,
                enable_centered_bias=False
                #, gradient_clip_norm=1 # helper functions that let you apply L2 norms (tf.clip_by_global_norm)
                )

print(model_dir)


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Exclude the encoded categorical & include the raw categorical features
Can you write Python code for it?

excludes = set([f for f in features_names_selected for f_cat in features_types_group["CATEGORICAL"] if f.startswith(f_cat)])
features_names_selected_raw = [f for f in features_names_selected if f not in excludes]
features_names_selected_raw = list(features_types_group["CATEGORICAL"]) + features_names_selected_raw

print("Exclude encoded categorical: ", excludes)
print("Include raw categorical: ", features_types_group["CATEGORICAL"])


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Initialise the perfromance statistics output
Can you write Python code for it?


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Represent the input data as the fundamental unit of TensorFlow computations
Can you write Python code for it?

def input_fn(file_names, batch_size, num_epochs, randomize_input):
    df_dict = read_csv_batches(file_names, batch_size, features_headers, num_epochs, randomize_input)

    with tf.Session(config=config) as sess:
        # Creates a dictionary mapping from each continuous feature column name (k) to
        # the values of that column stored in a constant Tensor.
        continuous_cols = {k: tf.string_to_number(df_dict[k], out_type=tf.float32)
                           for k in continuous_features}

        # Creates a dictionary mapping from each categorical feature column name (k)
        # to the values of that column stored in a tf.SparseTensor.
        categorical_cols = {
            k: tf.SparseTensor(
                indices=[[i, 0] for i in range(int(df_dict[k].get_shape()[0]))],
                values=df_dict[k],
                dense_shape=[int(df_dict[k].get_shape()[0]), 1])
            for k in discrete_features}

        # Merges the two dictionaries into one.
        feature_cols = {**continuous_cols, **categorical_cols}

        # Converts the label column into a constant Tensor.
        label = tf.string_to_number(df_dict[target_feature], out_type=tf.int32)

    # Returns the feature columns and the label.
    return feature_cols, label


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Save the output summaries
Can you write Python code for it?

summaries["train"] = dict()
summaries["train"]["results"] = results
summaries["train"]["predict_proba"] = model_dnn.predict_proba(
    input_fn=lambda: input_fn(train_file_names_full, test_batch_size,
                              test_num_epochs, test_randomize_input))


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Select features and save to CSV
Can you write Python code for it?

# save train sample
readers_writers.save_csv(
    data=pd.concat([features["train_target"].loc[:, [target_feature]],
                    features["train_indep"].loc[:, features_names_selected_raw]], axis=1),
    path=CONSTANTS.io_path, title=train_file_names[0], append=False)
print("File size: ", os.stat(os.path.join(CONSTANTS.io_path, train_file_names[0] + ".csv")).st_size)

# save test sample
readers_writers.save_csv(
    data=pd.concat([features["test_target"].loc[:, [target_feature]],
                    features["test_indep"].loc[:, features_names_selected_raw]], axis=1),
    path=CONSTANTS.io_path, title=test_file_names[0], append=False)
print("File size: ", os.stat(os.path.join(CONSTANTS.io_path, test_file_names[0] + ".csv")).st_size)


I am working on the file "TCARER_TensorFlow.ipynb".
The first task is:
Select the top N features
Can you write Python code for it?

top_n_features = 300

features_names_selected_raw = features_names_selected_raw[0:top_n_features]