I am working on the file "RandomizedAlgorithms/CW2-2 Shuffling.ipynb".
The first task is:
From the charts above, we can see that the Fisher Yates Shuffle is the best shuffling algorithm: it produces uniformly random permutations.
Then we have several not uniform algorithms:

Fisher Yates Buggy 1: We can see that is generally uniform, apart from the diagonal. This is because an element initially in position i never ends up in position i after the swap.

Random Sort Shuffling: The chart looks extremely dark, but we can spot a very bright point that corresponds to the position (0, 0), and in general elements close to the diagonal are lighter than elements further away. This is because, for an element in order to be swapped many times, it needs to get the same result from the random comparator again and again, which is unlikely.

Fisher Yates Buggy 2: Again this chart is quite difficult to read since it's almost completely dark blue, but we can spot a very bright element in position (n, n), meaning that the last element rarely changes its position. This is because the loop doesn't include this element, and therefore it can be swapped only if another element gets its index from the random generator, whereas other elements gets quite likely swapped also when it's their turn in the loop. Since that last element has such a high frequence of appearence in the last position, others values, even if close between them, all appear blue being so small comparared to that frequence.

Additional Algorithms
Can you write Python code for it?

random_sort_uniform_swap_matrix = swap_matrix(random_sort_shuffle_uniform, n_of_elems, times)

permute_with_all_swap_matrix = swap_matrix(permute_with_all, n_of_elems, times)

permute_by_cyclic_swap_matrix = swap_matrix(permute_by_cyclic, n_of_elems, times)

additional_matrixes = [
            'random_sort_uniform_swap_matrix',
            'permute_with_all_swap_matrix',
            'permute_by_cyclic_swap_matrix',
           ]

plot_matrixes(additional_matrixes)


I am working on the file "RandomizedAlgorithms/CW2-2 Shuffling.ipynb".
The first task is:
Other buggy implementations
Can you write Python code for it?

# Choose the element to swap from the whole array at every iteration
# Iterate forward
# Permute-With-All from Cormen
def permute_with_all(a):
    n = len(a)
    for i in range(n):
        j = random.randint(0, n - 1)
        swap(a, i, j)

# Choose a random offset and then shift the array with that offset
# Permute-By-Cyclic from Cormen
def permute_by_cyclic(a):
    n = len(a)
    offset = random.randint(0, n - 1)
    b = a[-offset:] + a[:-offset]

    # Only way to actually change the values in a also outside this function
    for i in range(n):
        a[i] = b[i]


I am working on the file "RandomizedAlgorithms/CW2-2 Shuffling.ipynb".
The first task is:
Random Shuffling
random_sort_shuffle
The random_sort_shuffle function should follow the idea of using a random comparator for sorting the array a.
Can you write Python code for it?

from functools import cmp_to_key

def comparator(a, b):
    return random.randint(-1, 1)

def random_sort_shuffle(a):
    a.sort(key=cmp_to_key(comparator))


I am working on the file "RandomizedAlgorithms/CW2-2 Shuffling.ipynb".
The first task is:
fisher_yates_shuffle
Can you write Python code for it?

def swap(a, i, j):
    a[i], a[j] = a[j], a[i]

def fisher_yates_shuffle(a):
    n = len(a)
    for i in range(n - 1):
        j = random.randint(i, n - 1)
        swap(a, i, j)


I am working on the file "Range().ipynb".
The first task is:
Great! Notice how it went up to 20, but doesn't actually produce 20. Just like in indexing. What about step size? We can specify that as a third argument:
Can you write Python code for it?


I am working on the file "RapWordsTalk.ipynb".
The first task is:

Can you write Python code for it?

lyrics_split = [line for line in df_data["lyrics"][23375].split("\n") if len(line) > 0]
print(lyrics_split[10:-30])

target_word = "trifle"

def get_rhymes(lyrics_split):
    lines_with_rhyming_parts = list()
    for line in lyrics_split:
        words = line.split()
        last_word = words[-1].strip('.,?!;:') # .strip() to remove any punctuation
        last_word_p = pronouncing.phones_for_word(last_word)

        if len(last_word_p) > 0:
            if len(last_word_p) > 1:
                last_word_p = [last_word_p[0],]

            for phones in last_word_p:
                rhyming_part = pronouncing.rhyming_part(phones)
                line_with_part = [rhyming_part[:2], line]
                #print(line_with_part)
                lines_with_rhyming_parts.append(line_with_part)
        else:
            line_with_part = ["N/A", line]
            lines_with_rhyming_parts.append(line_with_part)

    return lines_with_rhyming_parts

rhyme_lines = get_rhymes(lyrics_split)

rhyme_lines[40:60]

import itertools
import operator

grouped_rhymes = []
for key,group in itertools.groupby(rhyme_lines, operator.itemgetter(0)):
    merged_group = [g[1] for g in group]
    grouped_rhymes.append(list(merged_group))

grouped_rhymes[15:35]

def rhymegroup(target_word, grouped_rhymes):
    group_index_of_target = -1

    for groupid, rhymes in enumerate(grouped_rhymes):
        #print(groupid, rhymes)
        for line in rhymes:
            if target_word in line:
                group_index_of_target = groupid

    if group_index_of_target != -1:
        return grouped_rhymes[group_index_of_target]
    else:
        return "N/A"

rhymegroup("trifle", grouped_rhymes)

def rhymegroup_from_word(word, lyrics):
    lyrics_split = [line for line in lyrics.split("\n") if len(line) > 0]
    grouped_rhymes = []
    for key,group in itertools.groupby(rhyme_lines, operator.itemgetter(0)):
        merged_group = [g[1] for g in group]
        grouped_rhymes.append(list(merged_group))
   ...


I am working on the file "RapWordsTalk.ipynb".
The first task is:
Checking Words
Can you write Python code for it?

for artist, song, lyrics in zip(df_data["artist"],
                                df_data["song"],
                                df_data["lyrics"]):
    if "python" in set(lyrics.split()):
        print(artist, " - ", song)

for artist, song, lyrics in zip(df_data["artist"],
                                df_data["song"],
                                df_data["lyrics"]):
    if "anaconda" in set(lyrics.split()):
        print(artist, " - ", song)


I am working on the file "RapWordsTalk.ipynb".
The first task is:
Extract Rhyming Couplet
Can you write Python code for it?

import pronouncing

pronouncing.rhymes("star")[0:20]

print(pronouncing.phones_for_word("high"))
print(pronouncing.phones_for_word("sky"))

print(pronouncing.phones_for_word("orange"))
print(pronouncing.phones_for_word("hinge"))

def rhymes_per_line(lyrics_split):
    rhymes = []
    for line in lyrics_split:
        words = line.strip().split()
        last_word = words[-1].strip('.,?!;:')
        last_word_p = pronouncing.phones_for_word(last_word)
        if len(last_word_p) > 0:
            rhymes.append((pronouncing.rhyming_part(last_word_p[0]),line))
    return rhymes

rhymes_per_line(lyrics_split)


I am working on the file "RapWordsTalk.ipynb".
The first task is:
Extract Surrounding Lines
Can you write Python code for it?

lyrics_split=["Twinkle, twinkle, little star",
"How I wonder what you are",
"Up above the world so high",
"Like a diamond in the sky"]

def get_rhymegroup(target_word, lyrics_split):
    group_index_of_target = -1

    for groupid, line in enumerate(lyrics_split):
        if target_word in line:
            group_index_of_target = groupid

    if group_index_of_target > 0 and group_index_of_target < len(lyrics_split)-1:
        return lyrics_split[group_index_of_target-1:group_index_of_target+2]
    else:
        return "N/A"

get_rhymegroup("high", lyrics_split)


I am working on the file "RapWordsTalk.ipynb".
The first task is:
Finding 'Good' Songs
Can you write Python code for it?

import spotipy
sp = spotipy.Spotify()

def test_track_search(sp, search_str):
    results = sp.search(q=search_str, type='track', limit=1)
    if len(results['tracks']['items']) > 0:
        print(results['tracks']['items'][0]['artists'][0]['name']," - ",
              results['tracks']['items'][0]['name'],
              results['tracks']['items'][0]['popularity'],)
    else:
        print("")

test_track_search(sp, 'Nas Memory Lane')


I am working on the file "RapWordsTalk.ipynb".
The first task is:
Finding 'Good' Songs with YouTube
Can you write Python code for it?

# Activate Google Data API
DEVELOPER_KEY = ""

from apiclient.discovery import build
from datetime import datetime

def youtube_search(q):
  youtube = build("youtube", "v3", developerKey=DEVELOPER_KEY)

  search_response = youtube.search().list(
    q=q, type="video",
    part="id,snippet", maxResults=1
  ).execute()

  for search_result in  search_response.get("items", []):
      if search_result is not None:
          video_id = search_result["id"]["videoId"]
          date_posted = search_result["snippet"]["publishedAt"]
          results = youtube.videos().list(
                    part="statistics", id=video_id
                    ).execute()
  return (video_id,
          float(results["items"][0]["statistics"]["viewCount"]),
          float((datetime.now()-datetime.strptime(date_posted, "%Y-%m-%dT%H:%M:%S.000Z")).days))

youtube_data = youtube_search("Nas Memory Lane")
print(youtube_data)

from IPython.display import HTML
HTML('<iframe width="560" height="315" src="https://www.youtube.com/embed/JXBFG2vsyCM?rel=0&controls=0&showinfo=0" frameborder="0" allowfullscreen></iframe>')


I am working on the file "RapWordsTalk.ipynb".
The first task is:
Finding Rare Rap Words
Can you write Python code for it?

from collections import Counter
rap_onegrams = Counter()

for lyrics in df_data["lyrics"]:
    rap_onegrams.update(lyrics.split())

rap_onegrams_df = pd.DataFrame.from_dict(rap_onegrams, orient='index').reset_index()
rap_onegrams_df.columns = ["word","count"]

rap_onegrams_df.sort_values(by="count").tail(n=20)

raptwl_df = pd.merge(twl_wordlist, rap_onegrams_df, on="word")
raptwl_df.head()

all_bigrapwords = set(raptwl_df[raptwl_df["count"] < 5]["word"].values)
len(all_bigrapwords)

df_data["rap_matches"] = []
for lyrics in df_data["lyrics"]:
    found_words = set(lyrics.split()) & all_bigrapwords
    df_data["rap_matches"].append(found_words)

rap_data = pd.DataFrame(df_data)
rap_data = rap_data[rap_data.rap_matches != set()]
rap_data

#rap_data[rap_data["artist"].str.startswith("Kanye")]
#rap_data[rap_data["artist"].str.startswith("Wu-Tang") & rap_data["filename"].str.contains("enter")]
rap_data[rap_data["artist"].str.startswith("Nas") & rap_data["filename"].str.contains("illmatic")]

# Search and replace 23375 to 1173 in the lyrics subset on Github!
print(rap_data.loc[23375]["matches"])
print(rap_data.loc[23375]["rap_matches"])

word = "trifle"


I am working on the file "RapWordsTalk.ipynb".
The first task is:
Finding Rare Words
Can you write Python code for it?

# The 1/3 million most frequent words, all lowercase, with counts.
# http://norvig.com/ngrams/
ngrams=pd.read_csv("assets/count_1w.txt",sep="\t",names=["word","count"])
ngrams.tail(n=20)

# The Tournament Word List (178,690 words) -- used by North American Scrabble players.
# http://norvig.com/ngrams/
twl_wordlist=pd.read_csv("assets/TWL06.txt",names=["word"])
twl_wordlist=pd.DataFrame(twl_wordlist["word"].str.lower())

twl_w_ngrams = pd.merge(twl_wordlist, ngrams)

twl_w_ngrams.sort_values(by="count").head(n=20)

np.sum(twl_w_ngrams["count"] < 400000)

all_bigwords = twl_w_ngrams[twl_w_ngrams["count"] < 400000]["word"].values
all_big_bigwords = [word for word in all_bigwords if len(word) > 4]
bigset = set(all_big_bigwords)

print(all_big_bigwords[:100])

df_data["matches"] = []
for lyrics in df_data["lyrics"]:
    found_words = set(lyrics.split()) & bigset
    df_data["matches"].append(found_words)

rap_data = pd.DataFrame(df_data)
rap_data = rap_data[rap_data.matches != set()]

rap_data.head(n=20)


I am working on the file "RapWordsTalk.ipynb".
The first task is:
Get Definitions from Context
Can you write Python code for it?

lyrics = rap_data.loc[23375]["lyrics"]
rap_sentence = [line for line in lyrics.split("\n") if word in line][0]
print(rap_sentence)

import nltk
split_sentence = nltk.word_tokenize(rap_sentence)
tagged_sentence = nltk.pos_tag(split_sentence,tagset="universal")
print(tagged_sentence)


I am working on the file "RapWordsTalk.ipynb".
The first task is:
Puttin' it Together
Can you write Python code for it?

print(rap_data.loc[23375])

word = "trifle"
artist = rap_data.loc[23375].artist
song = rap_data.loc[23375].song

part_of_speech, worddef = get_definition_with_lyrics(word,
                                                       rap_data.loc[23375].lyrics)
lyrics = rhymegroup_from_word(word, rap_data.loc[23375].lyrics)
youtube = youtube_search(artist + " " + song)[0]

slug = word+"-"+part_of_speech+"-"+artist

#print(word, part_of_speech, worddef[0], lyrics, artist, song, youtube)

post_body = post_template(word, part_of_speech, worddef[0], lyrics, artist, song, youtube)

user.create_text("rapwords",
                 format="html",
                 state="published",
                 slug=slug,
                 body=post_body,
                 tags=[part_of_speech],)


I am working on the file "Rapport.ipynb".
The first task is:
Après avoir trouvé la solution fondamentale du problème, nous avons commencé la résolution du problème de flambement de notre structure.
La résolution de ce problème consiste à trouver le coefficient multiplicateur de charges, $\lambda$.
Pour cela, on doit résoudre le problème :
$$
(K-\lambda G)U = 0
$$
avec $G$ la matrice de rigidité géométrique
On construit $G$ à l'aide la formule suivante:
$$
G^{(e)}{ij}=-\bar N_0^{(e)}\int_0^LS{1i}'(s)S_{1j}'(s) ds,\quad i,j=1,\ldots,6
$$
avec $N_0^{(e)}$ la valeur normale de l'effort pour un élément $e$
Nous avons, tout d'abord, récupéré les valeurs des efforts normaux pour chaque élément puis nous les avons mis dans un tableau.
Can you write Python code for it?

class Frame_Buckling1(LinearFrame):

    def N_local_stress(self,element):

        """
        Returns the normal force of an element.
        """
        Ke= self.K_local()
        Ue= self.U_e_local_coord(element)
        F=Ke*Ue
        N_local = F[3]
        return N_local

    def N_local_stress_tot(self):

        """
        Returns the normal force of all elements.
        """
        Ns=[self.N_local_stress(e)for e in range (self.nelements)]
        return Ns


I am working on the file "Rapport.ipynb".
The first task is:
Comme vous pouvez le constatez, la première valeur propre obtenue correspond aussi au premier coefficient multiplicateur de charges, $\lambda_{1}$. On a donc $\lambda_{1num}=8,7.10^{-3}$
Nous avons utilisé ce que nous avons vu en cours pour calculer la valeur théorique de $\lambda$ en utilisant les mêmes conditions que pour le modèle numérique.
Can you write Python code for it?


I am working on the file "Rapport.ipynb".
The first task is:
Nous appliquons ensuite à la matrice G les conditions de bords. Nous avons réutilisé le même principe que pour K.
Can you write Python code for it?

class Frame_Buckling(Frame_Buckling2):
    def bc_apply_G(self,G,blocked_dof):
        for (dof) in enumerate(blocked_dof):
            Gbc = G
            Gbc[dof, :] = 0
            Gbc[:, dof] = 0
            Gbc[dof, dof] = 1
        return Gbc


I am working on the file "Rapport.ipynb".
The first task is:
Nous avons appris lors de ces séances de projet que pour résoudre le problème $(K-\lambda G)U = 0$ il fallait obtenir les valeurs propres de la matrice $G^{-1}K$. Nous avons pour cela deux méthodes de résolution numériques, la méthode la puissance itérée et l'algorithme QR. Or, dans notre cas, la matrice G est singulière. Il nous est donc impossible de calculer un inverse.
Pour contourner ce problème, nous avons utilisé la bibliothèque sparse présente dans scipy.
Dans cette bibliothèque, il y a une fonction qui permet d'avoir les valeurs propres à partir d'une certaine sans sprendre en compte le caractère singulier de G.
Voici la résolution du problème aux valeurs propres:
Can you write Python code for it?

Ks = sparse.csr_matrix(K)
Gs = sparse.csr_matrix(G)
val, vect = linalg.eigsh(Ks, 5, Gs, which = 'LA', sigma =4.)
print(val)
frame.set_displacement(10*vect[:,0])
frame.plot_with_label()
frame.plot_displaced()


I am working on the file "Rapport.ipynb".
The first task is:
Nous avons obtenu que $\lambda_{1théo}=9.10^{-3}$.
Nous avons pu constater que notre valeur théorique et notre valeur numérique était semblable aux approximations près. Nous pouvons donc conclure que notre modèle numérique est bon et qu'il peut être utilisé pour calculer les structures qui nous intéressse.
3)Comparaison de notre modèle numérique avec notre modèle expérimentale
Après avoir validé notre modèle numérique, nous l'avons appliqué aux deux structures de l'exercice 2, soit le portique et le portique renforcé. Voici la discrétisation du problème :
Can you write Python code for it?

Lx=55. #en mm
Lyh=60. #en mm
Lyb=45. #en mm
nodes= np.array([[0.,0.],[0.,Lyb],[0.,Lyh+Lyb],[Lx/2,Lyh+Lyb],[Lx,Lyh+Lyb],[Lx,Lyb],[Lx,0.]])
elements=np.array([[0,1],[1,5],[1,2],[2,3],[3,4],[4,5],[5,6]])

frame= Frame_Buckling(nodes,elements)
frame.plot_with_label()


I am working on the file "Rapport.ipynb".
The first task is:
Pour le portique renforcé, nous avons appliqué une force de $1N$ au noeud 3 en y  et nous avons modifié le module de Young E des éléments 1,3 et 4 en le multipliant par 100.
Pour le portique, nous avons appliqué deux forces de $1N$ au noeud 2 et 4 en y.
Nous avons utilisé ensuite le même principe de résolution que pour le 2).
Pour le portique renforcé, nous avons obtenu un $\lambda_{num1}=4,0$.
Pour le portique, nous avons obtenu un $\lambda_{num2}=1,8$.
Dans les conditions prises pour le portique renforcé, il faudrait 400g au niveau du noeud 3 pour que la structure flambe.
Dans les conditions prises pour le portique, il faudrait 180g au niveau des noeuds 2 et 4  pour que la structure flambe.
Nous avons testé expérimentalement sur le portique et le portique pour voir si nos modèles numériques sont une modélisation acceptable de notre structure.
Voici la photo pour le portique renforcé, nous avons mis une masse totale de 400g au dessus de notre structure qui corresponderait à notre noeud 3 sur notre modélisation :

Pour vérifier le flambement de notre structure, nous avons appuyé doucement sur le côté de la structure et nous avons remarqué qu'il y avait un grand déplacement de notre structure ce qui n'était pas le cas quand nous mettions pas de masse.
Nous avons obtenu $\lambda_{exp1}=4,0$ pour le portique renforcé.
Nous avons fait de même pour le portique et nous avons trouvé $\lambda_{exp2}=1,7$.
Nous trouvons donc des valeurs valeurs proches entre notre modèle numérique avec $\lambda_{num1}=4,0$ et $\lambda_{num2}=1,8$ ,et notre modèle expérimental avec $\lambda_{exp1}=4,0$ et $\lambda_{exp2}=1,7$.
Conclusion
Pour conclure, nous avons pu comparer nos modèles expérimentaux et nos modèles numériques et voir qu'ils se comportaient de la même façon même si les mesures sur les modèles expérimentaux restent très approximatives.
Les approximations faites dans notre modèle numérique ne sont donc pas aberrantes.
Nous avons par la suite essayé notre modélisation numérique sur d'autres structures afin de savoir...
Can you write Python code for it?


I am working on the file "Rapport.ipynb".
The first task is:
Voici l'analyse du flambement de cette structure :
Can you write Python code for it?


I am working on the file "Rapport/Rapport.ipynb".
The first task is:

Rapport de stage

Robot controlé par DNF

Le but de ce stage consiste à utiliser des champs neuronaux dynamiques (ou DNF : Dynamic Neural Fields), c'est à dire plusieurs éléments permettant un calcul décentralisé et distribué avec un comportement émergent face à un stimulis extérieur, pour permettre l'élaboration d'un robot autonome capable d'éviter des obstacles.
Pour cette première expérience le modèle de robot utilisé sera un e-puck : un petit robot cylindrique avec des capteurs infrarouges tout autour de lui et une caméra. Les stimulis extérieurs seront donc pour cette première partie les capteurs de proximité infrarouge.
I. Théorie et calculs
1) DNF infrarouges
Les champs neuronaux dynamiques sont directement reliés aux capteurs infrarouges puis pour savoir quel angle semble le plus suceptible d'être un obstacle, on va utiliser l'activation du DNF qui va renvoyer 0 si rien n'est détecté ou une valeur positive sinon.
Sur le schéma suivant, l'intensité du capteur a été multiplié par 100. L'intensité du capteur est égal à la distance maximale de réception moins la distance actuele détectée. Elle varie donc de 0 à 0.04 m.
Can you write Python code for it?


I am working on the file "Rapport/Rapport.ipynb".
The first task is:
Pour vérifier le bon fonctionnement dynamique de la formule face à un obstacle, observons $\frac{d\psi}{dt} = \omega$ en fonction de $\psi$, l'angle du robot dans le repère absolu.
Nous connaissons la taille de l'axe des deux roues ($\Delta = 52 mm$) et le rayon des roues ($r = 20.5 mm$).
On en déduit que face à un obstacle $\omega =\frac{r}{\Delta}(v_d-v_g)= \frac{2r}{\Delta}v_d$ avec $v_d = -v_g$ la vitesse angulaire du moteur droit.
Et hors de l'obstacle $\omega = 0rad/s$ car les deux roues vont à la même vitesse.
On a donc
$$\omega = \frac{d\psi}{dt} =
\left {
   \begin{array}{r, c, l}
      \frac{4r}{\Delta (1+e^{(-\psi+\phi)*10} -1)} & si & x \in [-\frac{\pi}{2},\frac{\pi}{2}] \
      0 & sinon &
   \end{array}
\right .
$$
Sur le graphe ci-dessous qui reprèsente $\omega$, le point rouge correspond à un point fixe instable ou répulseur. En effet, à droite du point la dérivée est positive donc l'angle va augmenter et à gauche elle est négative donc l'angle va diminuer.
Can you write Python code for it?

def plotPsi(phi):
    x=np.linspace(-math.pi,math.pi)
    vR = 2/(1+np.exp((-x+phi)*10))-1
    for i in range(50):
        if x[i]<-math.pi/2+phi or x[i]>math.pi/2+phi:
            vR[i]=0
    r=20.5
    delta=52
    w=2*r/delta*vR
    plt.plot(x,w, label='Rotation')
    plt.scatter(phi,w[phi], 40, color ='red')
    plt.legend()
    plt.xlabel("Angle psi")
    plt.ylabel("Rotation du robot (rad/s)")
    plt.title("Dynamique selon l'obstacle situé à "+str(phi)+" rad")
    plt.xticks([-math.pi,-math.pi/2, 0, math.pi/2, math.pi],[r'$-\pi$', r'$-\frac{\pi}{2}$', r'$0$', r'$+\frac{\pi}{2}$', r'$+\pi$'])
    plt.show()

slider_phi = widgets.FloatSliderWidget(min=-math.pi/2, max=math.pi/2, step=0.1, value=0)
w=widgets.interactive(plotPsi,phi = slider_phi)
display(w)


I am working on the file "Rapport/Rapport.ipynb".
The first task is:
Sur le schéma ci-dessus, le nombre de capteurs est de 6 (les capteurs situés devant le robot E-Puck), pour avoir une plus grande utilité des DNF, nous avons modifié le robot virtuel pour avoir 50 capteurs. En réalité seulement la moitié est utilisée, ce qui permet ainsi au robot de revenir vers la cible dès qu'il n'a plus la cible devant.
2) DNF direction
Nous avons ensuite rajouté une cible dont le robot ne connait que la direction de la même manière que pour les DNF des capteurs infrarouges.
3) Navigation
Dans le but d'avoir un comportement intéressant face aux obstacles, nous avons décidé que la navigation serait régit par deux activations, l'une correspondant à la présence d'obstacle et l'autre la direction de la cible inhibé par la présence d'obstacle dans cette même direction.
Ainsi le robot va toujours en direction de la cible excepté dans le cas de la présence d'un obstacle entre eux deux, dans ce cas le robot tournera autour de l'obstacle jusqu'à retrouver la direction de la cible.
Nous avions aussi envisagé un déplacement de l'activation de la direction dans le cas d'un obstacle devant, mais cette manipulation demandant avec le logiciel DNFPY un mouvement très lent n'était pas envisageable dans ce cadre là bien que ce soit plus proche d'un comportement naturel.
5) Vitesse des roues
La formule permettant de calculer la vitesse se base sur les activations ci-dessus.
$v=\frac{\sum_{x=-\Pi}^{+\Pi}(Fd(x)activationN(x))+\sum_{x=-\Pi}^{+\Pi}(Fo(x)activationI(x))}{\sum_{x=-\Pi}^{+\Pi}activationI(x)+\sum_{x=-\Pi}^{+\Pi}activationN(x)}$
Avec $Fd = \frac{2}{1+e^{(\pm x)}}-0.5$ et $Fo = \frac{2}{1+e^{(\pm x*10)}}-1$.
Les fonction étant inverse selon le moteur correspondant de telle manière que le robot s'écarte des obstacles et se rapproche de la cible.
a) Face à un obstacle
Dans le cas d'un seul obstacle situé entre la cible et le robot, l'activation N étant inhibé, on peut simplifier la formule de cet manière...
Can you write Python code for it?

x, st = np.linspace(-math.pi/2,math.pi/2, retstep=True)
vL = 2/(1+np.exp(x*10))-1
vR = 2/(1+np.exp(-x*10))-1
plt.plot(x,vL,label='Vitesse roue gauche')
plt.plot(x,vR,label='Vitesse roue droite')
plt.legend()
plt.xlabel("Position de l'obstacle")
plt.ylabel("Vitesse angulaire (rad/s)")
plt.title("Vitesses selon l'angle de l'obstacle")
plt.xticks([-math.pi/2, 0, math.pi/2], [r'$-\frac{\pi}{2}$', r'$0$', r'$+\frac{\pi}{2}$'])
plt.show()


I am working on the file "Raspando dados na Web com Requests e BeautifulSoup.ipynb".
The first task is:
Agora chegou o momento de extra informacoes relevantes da página que acabamos de coletar, como o título da página
Can you write Python code for it?


I am working on the file "Raspando dados na Web com Requests e BeautifulSoup.ipynb".
The first task is:
Como podemos perceber ele retorna todo o conteúdo do título, mas é possível obter apenas a string do título da página
Can you write Python code for it?


I am working on the file "Raspando dados na Web com Requests e BeautifulSoup.ipynb".
The first task is:
Como podemos perceber esse método retorna todo conteúdo da página inclusive códigos Javascript ou CSS. Nao é exatamente o que gostariamos de obter, o texto da música está contido nessa sopa, mas nossa colher parece grande demais para come-la ;). Analisando a página que contém o texto da música temos uma boa pista. A tag < article > parece conter todo o conteúdo da letra da música. Vamos usar entao o método .findAll() passando como parâmetro a string 'article'
Can you write Python code for it?

article_tag = soup.find('article')
type(article_tag)

for br_tag in article_tag.find_all('br'):
    br_tag.extract()
text = article_tag.get_text(separator=u' ', strip=True)
print(text)


I am working on the file "Raspando dados na Web com Requests e BeautifulSoup.ipynb".
The first task is:
Muito bem, o que nós precisamos fazer agora é extrair o texto da música que é o nosso objetivo inicial, para isso o BeautifulSoup contém um método chamado .get_text()
Can you write Python code for it?

# Extrair o texto da nossa sopa e imprimir
text = soup.get_text()
#print(text)


I am working on the file "RaspberryPiProgramming.ipynb".
The first task is:
GPIO Digital input - 2  (Pull Down)
Raspberry Pi has internal Pull Up/Down register, which can be enabled with setup() function.

Pull down register is enabled with GPIO.setup() function
Can you write Python code for it?

import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)  # use GPIO numbering, see output of gpio command
GPIO.setup(18, GPIO.IN, pull_up_down=GPIO_PUP_DOWN) # PIN18: Input (Pull Down enabled)
if GPIO.input(18) == GPIO.HIGH:
    print('PIN18: HIGH')
else:
    print('PIN18: LOW')


I am working on the file "Rate Distribution.ipynb".
The first task is:
A key point to note here is that rates are implicitly time-weighted. If you were to fit line segments and make each segment an equal-contributor to the rate distribution, then the distribution would look quite different.
Distance-weighted rate distribution
Can you write Python code for it?

plt.hist(phi29results.rate[phi29results.rate > 0], bins =  30, weights = phi29results.rate[phi29results.rate > 0], normed = True)
plt.plot(x,y,color="red", lw = 4)
plt.title("distance-weighted rate distribution")
plt.xlabel("Rate nt/s")
plt.ylabel("Nucleotides synthesised (fractional)")


I am working on the file "Rate Distribution.ipynb".
The first task is:
In this case, the info we are interested in is the rate column - this has not had noise added to it. The rates are drawn from a normal distribution centred at 80 nt/s, with a standard deviation of 40 nt/s. Each row is implicitly one second, and the distance travelled in each time step is therefore numerically equal to the rate.
Time-weighted rate distribution
Make a normal distribution with sigma and mu the same as those specified in the simulator.
Can you write Python code for it?

def pdf(sigma, mu, x_range):
    x = x_range
    y = []
    for i in x:
        y.append((1/(math.sqrt(2*3.14159*sigma**2)))*(math.exp(-1*(((i - (mu))**2) / (2*sigma**2)))))
    return x,y

x,y = pdf(40, 80,np.arange(0,250,1.0))


I am working on the file "Rate Distribution.ipynb".
The first task is:
Perfect, but so far this is merely a tautology: Making the equivalent adjustment to the pdf and the histogram it describes means that they will continue to fit to each other. Let's summarise the the adjustment we have made in a function which describes our new pdf:
Can you write Python code for it?

def weighted_pdf(sigma, mu, x_range):
    x = x_range
    a = []
    for i in x:
        a.append(((i/(math.sqrt(2*3.14159*sigma**2)))*(math.exp(-1*(((i - (mu))**2) / (2*sigma**2))))))
        #Note that x is now in the numerator of the first part of the Gaussian function.
    y = []
    for i in a:
        y.append(i/sum(a))
    return x,y

x,y = pdf(40,80,np.arange(0,250,1.0))
x, z = weighted_pdf(40,80,np.arange(0,250,1.0))

plt.plot(x,y,color="red")
plt.plot(x,z,color="green")


I am working on the file "Rate Distribution.ipynb".
The first task is:
Plot the rate distribution and overlay the above pdf.
Can you write Python code for it?

plt.hist(phi29results.rate[phi29results.rate > 0], bins =  30, normed = True)
plt.plot(x,y,color="red", lw = 4)
plt.title("Time-weighted rate distribution")
plt.xlabel("Rate nt/s")
plt.ylabel("Time (fractional)")


I am working on the file "Rate Distribution.ipynb".
The first task is:
Simulation of a large number of events
Generate a large results table.
Can you write Python code for it?

phi29results = pd.DataFrame()
for j in range(0,1000):
    temp = trajectory_simulator(pre_duration = 300,
                         pre_sigma = 20,
                         post_duration = 250,
                         mean_duration = 100,
                         min_duration = 10,
                         mean_rate = 80,
                         rate_sigma = 40,
                         noise_sigma = 100,
                         noise_sigma_sigma = 20,
                         pause_prob = 0.1,
                         pause_duration_prob = 0.5,
                         rate_change_prob = 0.01,
                         DNA_length = 15000,
                         trajectory_number = j)
    phi29results = phi29results.append(temp)

phi29results.head()


I am working on the file "Rate Distribution.ipynb".
The first task is:
This fit is OK for a guess, but how can we get there from first principles? Let's multiply by x and normalise, i.e. the same thing we did to the histogram.
Can you write Python code for it?

x,y = pdf(40, 80,np.arange(0,250,1.0)) #regenerate the original pdf with sigma = 40 and mu = 80
a = []
for i in range(0,len(x)):
    a.append(y[i]*i) #Multiply y by x
asum = sum(a)
z = []
for i in a:
    z.append(i/asum) #Normalise

plt.plot(x,y,color="red")
plt.plot(x,z,color="green")

plt.hist(phi29results.rate[phi29results.rate > 0], bins =  30, weights = phi29results.rate[phi29results.rate > 0], normed = True)
plt.plot(x,z,color="green",lw=4)
plt.title("distance-weighted rate distribution")
plt.xlabel("Rate nt/s")
plt.ylabel("Nucleotides synthesised (fractional)")


I am working on the file "RatingAlgorithmComparison.ipynb".
The first task is:
1. Random baseline
We generate random ratings with a standard Gaussian variable. This will be the baseline for comparison.
Can you write Python code for it?

random_model = rating_utils.RandomRating()
random_model.fit(train, rating_utils.get_teams(data))
print random_model.ratings

plt.hist(random_model.ratings)
pass


I am working on the file "RatingAlgorithmComparison.ipynb".
The first task is:
2. Fit and test Massey's method
We fit and test Massey's method.
We calculate the ratings and plot out the distribution.
Can you write Python code for it?

massey_model = rating_utils.MasseyMethod()
massey_model.fit(train, rating_utils.get_teams(data))
print massey_model.ratings

plt.hist(massey_model.ratings)
pass


I am working on the file "RatingAlgorithmComparison.ipynb".
The first task is:
3. Fit and test Markov's method on matches
We fit and test Markov's method on match results.
We calculate the ratings and plot out the distribution.
Can you write Python code for it?

markov_match_model = rating_utils.MarkovMatchMethod()
markov_match_model.fit(train, rating_utils.get_teams(data))
print markov_match_model.ratings

plt.hist(markov_match_model.ratings)
pass


I am working on the file "RatingAlgorithmComparison.ipynb".
The first task is:
4. Fit and test Markov's method on score
We fit and test Markov's method on match score.
We calculate the ratings and plot out the distribution.
Can you write Python code for it?

markov_score_model = rating_utils.MarkovScoreMethod()
markov_score_model.fit(train, rating_utils.get_teams(data))
print markov_score_model.ratings

plt.hist(markov_score_model.ratings)
pass


I am working on the file "RatingAlgorithmComparison.ipynb".
The first task is:
5. Fit and test Colley's method
We fit and test Colley's method.
We calculate the ratings and plot out the distribution.
Can you write Python code for it?

colley_model = rating_utils.ColleyMethod()
colley_model.fit(train, rating_utils.get_teams(data))
print colley_model.ratings

plt.hist(colley_model.ratings)
pass


I am working on the file "RatingAlgorithmComparison.ipynb".
The first task is:
Data processing
We load the training data and split it into training and test set
Can you write Python code for it?

data = pd.read_csv('data/12-13.csv')

indices = np.random.rand(len(data)) < 0.7

train = data[indices].reset_index()
test = data[~indices].reset_index()

print rating_utils.get_teams(data)

print len(train), len(test)


I am working on the file "RatingAlgorithmComparison.ipynb".
The first task is:
We now compute the prediction accuracy and cross entropy loss
Can you write Python code for it?

print "Training and testing accuracy:"
print random_model.test_accuracy(train), random_model.test_accuracy(test)
print "\nTraining and testing cross entropy loss:"
print random_model.test_cross_entropy_loss(train), random_model.test_cross_entropy_loss(test)


print "Training and testing accuracy:"
print markov_match_model.test_accuracy(train), markov_match_model.test_accuracy(test)
print "\nTraining and testing cross entropy loss:"
print markov_match_model.test_cross_entropy_loss(train), markov_match_model.test_cross_entropy_loss(test)


print "Training and testing accuracy:"
print colley_model.test_accuracy(train), colley_model.test_accuracy(test)
print "\nTraining and testing cross entropy loss:"
print colley_model.test_cross_entropy_loss(train), colley_model.test_cross_entropy_loss(test)


I am working on the file "Raw Data/ERCOT/Hourly wind generation/.ipynb_checkpoints/Exploring hourly wind data-checkpoint.ipynb".
The first task is:
Wind as % of total ERCOT load
The figures below show average values of wind as a percent of total ERCOT load by hour in 2009 and 2015. This is a much bigger difference - 2015 values are about double 2009 values.
Can you write Python code for it?

sns.factorplot('hour', 'Wind % of ERCOT Load', data=df_2009, aspect=1.5)
plt.title('2009 Wind as fraction of total load')

sns.factorplot('hour', 'Wind Output, % of Load', data=df_2015, aspect=1.5)
plt.title('2015 Wind as fraction of total load')


I am working on the file "Raw Data/ERCOT/Hourly wind generation/.ipynb_checkpoints/Exploring hourly wind data-checkpoint.ipynb".
The first task is:
Wind capacity factor
The figures below show average CF of installed wind in 2009 and 2015. Looks like they got ~5 percentage points up over that time.
Can you write Python code for it?

sns.factorplot('hour', '% Installed Wind Capacity', data=df_2009, aspect=1.5)
plt.title('2009 Wind Capacity by Hour of Day')

sns.factorplot('hour', 'Wind Output, % of Installed', data=df_2015, aspect=1.5)
plt.title('2015 Wind Capacity by Hour of Day')


I am working on the file "Raw Data/ERCOT/Hourly wind generation/Exploring hourly wind data.ipynb".
The first task is:
Export the clean ERCOT wind and load data
Can you write Python code for it?

import os

filename = 'ERCOT wind data.csv'
path = '../../../Clean Data'
fullpath = os.path.join(path, filename)

df_final.to_csv(fullpath)


I am working on the file "Raw Data/ERCOT/Hourly wind generation/Exploring hourly wind data.ipynb".
The first task is:
Try reading in all .xls and .xlsx files
Can you write Python code for it?

import glob

files = glob.glob('*.xls')
files.extend(glob.glob('*.xlsx'))

# df = pd.DataFrame()
# for fn in files:
#     temp_df = pd.read_excel(fn, sn='numbers', index_col=0)
#     if temp_df.index.values[0] != pd.Timestamp:
#         print fn
#     try:

#     df = pd.concat[]

df = pd.concat([pd.read_excel(fn, sn='numbers', index_col=0) for fn in files])

df.sort_index(inplace=True)

df.head()

df.tail()

df = df.iloc[:-1,:]

df.tail()

cols = df_2009.columns[:-2]
cols

df_final = df.loc[:,cols]
df_final.head()

df_final.dtypes

df_final.plot(y='Total Wind Installed, MW', use_index=True)

sns.jointplot('ERCOT Load, MW', 'Total Wind Output, MW', data=df_final, kind='hex')


I am working on the file "Raw Data/ERCOT/Hourly wind generation/Exploring hourly wind data.ipynb".
The first task is:
Wind capacity factor
The figures below show average CF of installed wind in 2009 and 2015. Looks like they got ~5 percentage points up over that time.
Can you write Python code for it?


I am working on the file "Raw Data/Filter and clean raw data.ipynb".
The first task is:
ERCOT wind and load data
Find filenames for all .xls and .xlsx files in the wind generation subfolder
Can you write Python code for it?

path = 'ERCOT/Hourly wind generation'
full_xls = os.path.join(path, '*.xls')
full_xlsx = os.path.join(path, '*.xlsx')

files = glob.glob(full_xls)
files.extend(glob.glob(full_xlsx))

cols = ['ERCOT Load, MW', 'Total Wind Installed, MW',
       'Total Wind Output, MW', 'Wind Output, % of Installed',
       'Wind Output, % of Load', '1-hr MW change', '1-hr % change']

df = pd.concat([pd.read_excel(fn, sn='numbers', index_col=0) for fn in files])

df.sort_index(inplace=True)

df.head()

df = df.loc[:,cols]

df.head()

df.plot(y='Total Wind Installed, MW', use_index=True)


I am working on the file "Raw Data/Filter and clean raw data.ipynb".
The first task is:
Export clean ERCOT data
Can you write Python code for it?

filename = 'ERCOT wind data.csv'
path = '../Clean Data'
fullpath = os.path.join(path, filename)

df.to_csv(fullpath)


I am working on the file "Raw Data/Load fuel prices.ipynb".
The first task is:
Coal
Price is $/short ton
EIA data
Source
Can you write Python code for it?

path = os.path.join('EIA data', 'Coal prices', 'Texas electric sector coal price.xlsx')
coal_temp = pd.read_excel(path, sheetname='Sheet1')

coal_temp.head()

coal_temp.loc[:,'Quarter'] = coal_temp.loc[:,'Date'].apply(lambda x: int(x[1]))
coal_temp.loc[:,'Year'] = coal_temp.loc[:,'Date'].apply(lambda x: int(x[-4:]))

coal_temp.head()

for coal in ['All coal', 'Lignite', 'Subbituminous']:
    plt.plot(coal_temp.index, coal_temp[coal], label=coal)
plt.legend(loc=0)

coal_temp[['All coal', 'Lignite', 'Subbituminous']].corr()


I am working on the file "Raw Data/Load fuel prices.ipynb".
The first task is:
Combine fuel dataframes
Can you write Python code for it?

fuel_price = pd.merge(ng_price, coal_price, how='right', on=['Month', 'Year'])

fuel_price.head()

fuel_price.tail()

fuel_price.drop(['Date_x', 'Date_y', 'tempdate', 'Quarter'], axis=1, inplace=True)

path = os.path.join('..', 'Clean Data', 'Fuel prices.csv')
fuel_price.to_csv(path)


I am working on the file "Raw Data/Load fuel prices.ipynb".
The first task is:
Natural gas
Price is $/thousand cubic feet (mcf)
Can you write Python code for it?

path = os.path.join('EIA data', 'Natural gas price', 'N3045TX3m.xls')
ng_price = pd.read_excel(path, sheetname='Data 1', header=2)

ng_price.head()

ng_price.dtypes

ng_price.rename(columns={"Texas Natural Gas Price Sold to Electric Power Consumers (Dollars per Thousand Cubic Feet)":
                         "NG Price ($/mcf)"}, inplace=True)

ng_price.head()

ng_price.loc[:,'Month'] = ng_price.loc[:,'Date'].apply(lambda x: x.month)
ng_price.loc[:,'Year'] = ng_price.loc[:,'Date'].apply(lambda x: x.year)

ng_price.head()

ng_price.dtypes


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Column order doesn't match in all years
Can you write Python code for it?


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Combine all df's from the dict into one df
Concat all dataframes, reset the index, determine the primary fuel type for each facility, filter to only include fossil power plants, and export as a csv
Can you write Python code for it?

all923 = pd.concat(aggEIADict)

all923.head()

all923.reset_index(drop=True, inplace=True)

# Check column numbers to use in the function below
all923.iloc[1,1:27]

def top_fuel(row):
    #Fraction of largest fuel for electric heat input
    try:
        fuel = row.iloc[1:27].idxmax()[29:]
    except:
        return None
    return fuel

all923['FUEL'] = all923.apply(top_fuel, axis=1)

all923.head()

fossil923 = all923.loc[all923['FUEL'].isin(['DFO', 'LIG', 'NG', 'PC', 'SUB'])]


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Export EIA 860 data
Can you write Python code for it?

filename = 'EIA 860.pkl'
path = '../Clean Data'
fullpath = os.path.join(path, filename)

pickle.dump(eia860Dict, open(fullpath, 'wb'))


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Export EPA data as a series of dataframes
The whole dictionary is too big as a pickle file
Can you write Python code for it?

# pickle with gzip, from http://stackoverflow.com/questions/18474791/decreasing-the-size-of-cpickle-objects
def save_zipped_pickle(obj, filename, protocol=-1):
    with gzip.open(filename, 'wb') as f:
        pickle.dump(obj, f, protocol)

filename = 'EPA hourly dictionary.pgz'
path = '../Clean Data'
fullpath = os.path.join(path, filename)

save_zipped_pickle(epaDict, fullpath)

df = epaDict['2015 July-Dec']

df.head()

set(df['PLANT_ID'])

df_temp = df[df['PLANT_ID'].isin([127, 298, 3439])].fillna(0)

df_temp.head()

g = sns.FacetGrid(df_temp, col='PLANT_ID')
g.map(plt.plot, 'datetime', 'GROSS LOAD (MW)')
g.set_xticklabels(rotation=30)

path = os.path.join('..', 'Exploratory visualization', 'Midterm figures', 'Sample hourly load.svg')
plt.savefig(path)


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Export the EIA 923 data as pickle
Just sending the dictionary to a pickle file for now. At least doing this will save several min of time loading and processing the data in the future.
Can you write Python code for it?

filename = 'EIA 923.pkl'
path = '../Clean Data'
fullpath = os.path.join(path, filename)

pickle.dump(aggEIADict, open(fullpath, 'wb'))


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Export the EIA 923 data dataframe as csv
Export the dataframe with primary fuel and filtered to only include fossil plants
Can you write Python code for it?

filename = 'Fossil EIA 923.csv'
path = '../Clean Data'
fullpath = os.path.join(path, filename)
fossil923.to_csv(fullpath)


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Finally join the two data sources
Switch to an inner join?
No need to join. Can keep them as separate databases, since one is hourly data and the other is annual/monthly Create a clustering dataframe with index of all plant IDs (from the EPA hourly data), add columns with variables. Calculate the inputs in separate dataframes - example is to calculate ramp rate values in the EPA hourly data, then put the results in the clustering dataframe.
Can you write Python code for it?

#Join the two data sources on PLANT_ID
fullData = {key:df.merge(aggEIADict[df["YEAR"][0]], on="PLANT_ID") for key, df in epaDict.iteritems()}

fullData[fullData.keys()[0]].head()


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
First rename the column name so we can merge on that column, then change the datatype of date to a datetime object
Can you write Python code for it?

#Rename the column names to remove the leading space.
for key, df in epaDict.iteritems():
    colNames = [name.upper().strip() for name in df.columns]
    colNames[colNames.index("FACILITY ID (ORISPL)")] = "PLANT_ID"
    epaDict[key].columns = colNames

#Convert DATE to datetime object
#Add new column DATETIME with both date and hour
for key, df in epaDict.iteritems():
    epaDict[key]["DATE"] = pd.to_datetime(df["DATE"])
    epaDict[key]['DATETIME'] = df['DATE'] + pd.to_timedelta(df['HOUR'], unit='h')


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Get the max ramp rate for every plant for each year
Can you write Python code for it?

cols = ['PLANT_ID', 'YEAR', 'Gen Change']

ramp_rate_list = []
for year in ramp_df['YEAR'].unique():
    for plant in ramp_df.loc[ramp_df['YEAR']==year,'PLANT_ID'].unique():
        # 95th percentile ramp rate per plant per year
        ramp_95 = ramp_df.loc[(ramp_df['PLANT_ID']== plant) &
                              (ramp_df['YEAR']==year),'Gen Change'].quantile(0.95, interpolation='nearest')
        ramp_rate_list.append([plant, year, ramp_95])

ramp_rate_df = pd.DataFrame(ramp_rate_list, columns=['plant_id', 'year', 'ramp_rate'])

ramp_rate_df.describe()

for key, df in clusterDict.iteritems():
    clusterDict[key] = pd.merge(clusterDict[key], ramp_rate_df, how='left', on=['plant_id', 'year'])

clusterDict[2010].head()

# Check plants larger than 25MW, which is the lower limit for EPA
clusterDict[2010][clusterDict[2010].capacity >=25].describe()

for key in clusterDict.keys():
    print key, clusterDict[key].plant_id.count(), clusterDict[key].ramp_rate.count()


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Loading EIA 860 Data
Can you write Python code for it?

# Iterate through the directory to find all the files to import
path = os.path.join('EIA Data', '860-No_Header')
full_path = os.path.join(path, '*.*')

eia860Names = os.listdir(path)

# Rename the keys for easier merging later
fileName860Map = {  'GenY07.xls':2007,
                    'GenY08.xls':2008,
                    'GeneratorY09.xls':2009,
                    'GeneratorsY2010.xls':2010,
                    'GeneratorY2011.xlsx':2011,
                    'GeneratorY2012.xlsx':2012,
                    '3_1_Generator_Y2013.xlsx':2013,
                    '3_1_Generator_Y2014.xlsx':2014,
                    '3_1_Generator_Y2015.xlsx':2015}

#Load the files into data frames, one df per file
eia860Dict = {fileName860Map[fn]:pd.read_excel(os.path.join(path, fn)) for fn in eia860Names}

#Dict of values to replace to standardize column names across all dataframes
renameDict = {  "PLNTCODE":"PLANT_ID",
                "PLANT_CODE":"PLANT_ID",
                "Plant Code":"PLANT_ID",
                "NAMEPLATE":"NAMEPLATE_CAPACITY(MW)",
                "Nameplate Capacity (MW)":"NAMEPLATE_CAPACITY(MW)"}

#Replace the column name
def rename860(col):
    for old, new in renameDict.iteritems():
        col = col.replace(old, new)
    return col

#Iterate through each column name of each dataframe to standardize and select columns 'PLANT_ID', 'NAMEPLATE_CAPACITY(MW)'
for key, df in eia860Dict.iteritems():
    colNames = [rename860(col) for col in df.columns]
    eia860Dict[key].columns = colNames
    eia860Dict[key] = eia860Dict[key][["PLANT_ID", "NAMEPLATE_CAPACITY(MW)"]]

# verify the tables
for key, df in eia860Dict.iteritems():
    print key, df.columns, len(df)

# Iterate through each dataframe to perform aggregation by PLANT_ID
for key, df in eia860Dict.iteritems():
    gb = df.groupby(by='PLANT_ID').apply(lambda x: x['NAMEPLATE_CAPACITY(MW)'].sum())
    eia860Dict[key]['NAMEPLATE_CAPACITY(MW)'] = eia860Dict[key].PLANT_ID.apply(gb.get_value)
    eia860Dict[key] =...


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Loading the EPA Data, the path may need to be updated...
Can you write Python code for it?

#Read the EPA files into a dataframe
path2 = os.path.join('EPA air markets')
epaNames = os.listdir(path2)
filePaths = {dn:os.path.join(path2, dn, "*.txt") for dn in epaNames}
filePaths = {dn:glob.glob(val) for dn, val in filePaths.iteritems()}
epaDict = {key:pd.read_csv(fp, index_col = False) for key, val in filePaths.iteritems() for fp in val}


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Merge all epa files in one df
Can you write Python code for it?

columns = ['PLANT_ID', 'YEAR', 'DATE', 'HOUR', 'GROSS LOAD (MW)']
counter = 0
for key, df in epaDict.iteritems():
    if counter == 0:
        result = epaDict[key][columns]
        counter = 1
    else:
        result = result.append(epaDict[key][columns], ignore_index=True)

# Change nan to 0
result.fillna(0, inplace=True)

result.describe()


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
Save dict to csv
Can you write Python code for it?

# re-arrange column order
columns = ['year', 'plant_id', 'capacity', 'capacity_factor', 'efficiency', 'ramp_rate', 'fuel_type']

filename = 'Cluster_Data_2.csv'
path = '../Clean Data'
fullpath = os.path.join(path, filename)

counter = 0
for key, df in clusterDict.iteritems():
    # create the csv file
    if counter == 0:
        df[columns].sort_values(by='plant_id').to_csv(fullpath, sep=',',  index = False)
        counter += 1
    # append to existing csv file
    else:
        df[columns].sort_values(by='plant_id').to_csv(fullpath, sep=',',  index = False, header=False, mode = 'a')


I am working on the file "Raw Data/Merging EPA and EIA.ipynb".
The first task is:
The excel documents have different column names so we need to standardize them all
Can you write Python code for it?

#Dict of values to replace to standardize column names across all dataframes
monthDict = {"JANUARY":"JAN",
           "FEBRUARY":"FEB",
           "MARCH":"MAR",
           "APRIL":"APR",
           "MAY":"MAY",
           "JUNE":"JUN",
           "JULY":"JUL",
           "AUGUST":"AUG",
           "SEPTEMBER":"SEP",
           "OCTOBER":"OCT",
           "NOVEMBER":"NOV",
           "DECEMBER":"DEC"}

replaceDict = {"ELECTRIC":"ELEC",
               "&":"AND",
               "I.D.":"ID",
               "MMBTUPER":"MMBTU_PER"}

#Add "MMBTUMON" : "MMBTU_MON" to be replaced
for month in monthDict.values():
    replaceDict["MMBTU"+month] = "MMBTU_" + month

#Replace the column name
def rename(col):
    for old, new in monthDict.iteritems():
        col = col.replace(old, new)

    for old, new in replaceDict.iteritems():
        col = col.replace(old, new)

    col = col.replace("MMBTUS", "MMBTU")
    return col

#Iterate through each column name of each dataframe to standardize
for key, df in eiaDict.iteritems():
    colNames = [name.replace("\n", "_").replace(" ", "_").strip().upper() for name in df.columns]
    colNames = [rename(col) for col in colNames]
    eiaDict[key].columns = colNames


I am working on the file "Raw/Mitha/COMPUTING WITH TENSORFLOW.ipynb".
The first task is:
INTRODUCING PLACEHOLDERS
They are used to provide input whenever tensor flow is made to run computation. To mention a dimension of any length, None can be specified in the shape argument. The below example which deals with matrix multiplication of tensors-generated using the random function, provides an understanding of working with placeholder values.
Example 6:
Can you write Python code for it?

# The objective of this program is to familiarise with the use of placeholders and
# a few matrix arithmetic functions
# Using placeholder value inidicates directing the tensor
# value into Operations in graph.
# Placeholders should always fed with feed_dict argument while using Session.run()/tensor.eval()/operation.run() command
# Takes arguments: dtype, shape(optional),name(optional)
# the dtype elements indicate the type of tensor. For eg: float32, complex64, int8, qint8, bool, string etc
import tensorflow as tf
import numpy as np
val1 = tf.placeholder(tf.float32,shape = (8,8))
val2 = tf.placeholder(tf.float32, shape = (8,8))
randomNos2 = np.random.rand(8,8)
# tf.random_normal : Generates random numbers following normal distribution with mean =0.0, stddev = 1.0, dtype = tf.float32
randomNormal = tf.Variable(tf.random_normal([8,8]),name = 'NormalDistribution')
# Below command creates an OP (Operation) of type MatMul;
product = tf.matmul(val1,val2)
matInv = tf.matrix_inverse(product, name="MatrixInverse")
matDeter = tf.matrix_determinant(val1,name ="MatrixDeterminant")
eigenVec = tf.self_adjoint_eig(randomNormal, name = "EigenVectors")
# Unless specified, both val1 and val2 has to be of placeholder type input or they both
# have to be of Variable type which doesnt require initialization using feed_dict
# addition = tf.add(randomNormal,val1) # This command will give an invalid argument error as val1
#randomNormal are variable and placeholder values respectively.
init = tf.initialize_all_variables()
# With the below commands, the Op gets executed
with tf.Session() as sess:
        sess.run(init)
        print (sess.run(product,feed_dict={val1:randomNos2, val2:randomNos2}))
        print "INVERSE: ",(sess.run(matInv,feed_dict={val1:randomNos2, val2:randomNos2}))
# get_shape(): Returns the shape of a tensor; which would be either Fully-known shape(size and number of dimensions are known) or
# Partially-known shape(number of dimensions known but size is unknown) or Unknown Shape(as the name says, number of...


I am working on the file "Raw/Mitha/COMPUTING WITH TENSORFLOW.ipynb".
The first task is:
Tensor Board provides a visualization of the model that you create. It deals with 2 types of connections: Control dependency(denoted using dotted lines) and Data dependency(denoted using solid lines).
To record the variations produced by a particular function, provide them as inputs to scalar_summary ops with a tag name. Histogram_summary provides the distribution of an output variable from a layer.
tf.merge_all_summaries will merge the summaries created within the program, which is then directed to a summary_writer. Specify a logdir for summary writer to write the events.
To get the tensor board launched as below:
- run the command :
        tensorboard --logdir = /tmp/mnist_logs
- Open a browser and navigate to
        http:<server>:6006

Example 5:
Consider a simple model which calculates a new state using the error obtained from the previous state.The functions add, sub forms a node in the graph with their own definite operations. They produce zero or more tensors as inputs for the next set of nodes:
Can you write Python code for it?

#```python
import tensorflow as tf
# Variables: Memory Buffers, with tensors that update and hold values
# Needs explicit initializations
# tf.Variables creates a variable with value 1.
currState = tf.Variable(1)
constValue = tf.constant(5)
cProd = currState *5
#Subtraction operation: cProd - currState
interValue = tf.sub(cProd,currState)
#Addition
newState = tf.add(currState,interValue)
#The below operation is performed once the run command is executed
update = tf.assign(currState,newState)
#Whenever a variable is created, it is an empty node. It is only by initializing them the
# variable gets filled with the content, which is a value
init_op = tf.initialize_all_variables()
#Within Session class, the operations are executed and datas in the form of
#tensors are evaluated
with tf.Session() as sess:
        sess.run(init_op)
        #The below statement will execute to generate the currState value
        print(sess.run(currState))
        for _ in range(5):
                sess.run(update)
                #print(sess.run(currState))
                print (sess.run([currState, interValue]))

#```


I am working on the file "Raw/Mitha/COMPUTING WITH TENSORFLOW.ipynb".
The first task is:
Tensor dimensionality is denoted using shape, rank and dimension number. Tensor with rank 0 is a scalar. A rank 1 tensor is a vector. A rank 2 tensor is a matrix, ie you can access element using 2 variables like tensor[i,j]; here the dimension is 2D. In the case of rank 3, access it using tensor[i,j,k];here dimension is 3D.
The tensor elements can be subjected to slice, tile operations. The below example deals with slice command.
Example 2:
Can you write Python code for it?

import tensorflow as tf
import numpy as np
val1 = tf.placeholder(tf.float32,shape = (8,8))
val2 = tf.placeholder(tf.float32, shape = (8,8))
randomNos2 = np.random.rand(8,8)
product = tf.matmul(val1,val2)
inpSlice = tf.constant([[[1,11,111],[2,22,222]],
                  [[3,33,333],[4,44,444]],
                  [[5,55,555],[6,66,666]]])

init = tf.initialize_all_variables()
# Softplus activation function computes log(exp(featurePoints)+1)
softPlus = tf.nn.softplus(product)
with tf.Session() as sess:
        sess.run(init)
        print (sess.run(product,feed_dict={val1:randomNos2, val2:randomNos2}))
        result = sess.run(softPlus,feed_dict={val1:randomNos2, val2:randomNos2})
        # The slicing function takes the arguments input, size and begin
        print (result)
        print "SLICING ON TENSORS: "
        print "Applying slicing on softplus output",(sess.run(tf.slice(result, [1, 1], [1, 1])))

        print "Access1: ",(sess.run(tf.slice(inpSlice,[0,0,0],[1,1,1])))
        print "Access2: ",(sess.run(tf.slice(inpSlice,[0,0,0],[1,1,2])))
        print "Access3: ",(sess.run(tf.slice(inpSlice,[0,0,0],[1,1,3])))
# Interpretation of slice arguments:
# The begin parameter begin(i,j,k): will enable access to 'i' ranks; spanning 'j' dimensions;
# the number of values returned is 'k'
# This will access elements at the 2ranks, from both dimensions- starting from the range 0 as size is specified as 0

        print "Access7: ",(sess.run(tf.slice(inpSlice,[0,0,0],[2,1,1])))
        print "Access8: ",(sess.run(tf.slice(inpSlice,[0,0,0],[3,2,1])))


I am working on the file "Raw/Mitha/INTRODUCTION TO COMPUTING WITH TENSORFLOW.ipynb".
The first task is:
ACTIVATION FUNCTIONS
In the biological world, Neurons does the decision making process. The below figure shows a 3 layer neural network: with 2 hidden layers and one output layer. Each element in the input layer(Blue layer), hidden layer(yellow layer) and output layer(green layer) are considered as neurons that communicates between the layers.

<center> 3 Layer Neural Network</center>
A Perceptron neuron is an artificial neuron, that takes many binary inputs and gives binary output. To compute result, weights (real numbers indicating the role of the inputs in the corresponding output) were used. The weighted sum of the product of input and weights, added with bias are compared with a threshold value to decide the neuron output as 0 or 1.
The output from one layer is fed as input to the next layer. Such a network is called as a feed forward network. Each element in the hidden and output layer is called a neuron and it just does some simple tasks like: Reads input data, Process it and gives the output. By having a network of such a model we make it to ‘learn’, producing intelligent results that can be converged towards a decision making process.
Perceptron is a basic network model which gets a weighted input from the previous layer. The results from a perceptron is found to be flipping as the weights / bias values change.
Sigmoid neurons are considered to be better than perceptron. Though with some similarities with perceptron, sigmoid neurons deals with exponent values of inputs between 0 and 1. The output is a real number between 0 and 1. Sigmoid neuron is a smoothed version of a perceptron.
There are many other learning algorithms and they will make our model learn and adapt the weights and biases of our model so as to provide us with the decision made.
The activation functions produces a tensor of the same shape as their inputs.
The most common activation function is called the Sigmoid.
$$\sigma(x) = 1/(1+e^{-x})$$
Other activation functions include:Softplus, Sigmoid, tanh, elu and tensor flow have...
Can you write Python code for it?

import tensorflow as tf
import numpy as np

input = tf.placeholder("float",[1,3])
weights = tf.Variable(tf.random_normal([3,3]),name = "Weights")
y = tf.matmul(input,weights)
relu_output = tf.nn.relu(y)
softmax = tf.nn.softmax(relu_output)
init = tf.initialize_all_variables()

with tf.Session() as sess:
    sess.run(init)
    print "Matrix PRoduct: ",sess.run(y,feed_dict={input:np.array([[1.0,2.0,3.0]])})
    print "Weights: ",sess.run(weights)
    print "Applying Activation function Softmax: ",sess.run(softmax,feed_dict={input:np.array([[1.0,2.0,3.0]])})


I am working on the file "Raw/Mitha/INTRODUCTION TO COMPUTING WITH TENSORFLOW.ipynb".
The first task is:
Example 3:
 Tensor flow offers a variety of basic arithemtic functions to do computations.
 A few of them are implemented below for ease.
``` python
The following program shows the basic arithemtic operations using Tenspr Flow and the #  arguments required to be passed.
Takes arguments: x input,y input, name=None (optional; without space)
The x input and y input should be tensors;Must be one of the types: float32, float 64, uint8, int16, int32, int 64, complex64
y should be of the same type as x input; variables are case sensitive
import tensorflow as tf
Input constant float values.
a = tf.constant([[4.0, 8.0,10.0],[20.0,40.0,80.0]])
b = tf.constant([[8.0, 16.0,15.0],[25.0,40.0,50.0]])
The common arithemtic operators can be implemented as below:
sumVal = tf.add (a,b, name = "AdditiveOp")
TrueDiv function produces floating point quotients; Here integer operands are converted to floating point values
If integer values of type int8 or int16: Casted to float32 / If integer values of type int32 or int64: Casted to float64
If a and b are of different types, it gives error
QuoValtrue = tf.truediv(a,b)
The cross function provides the cross product of 2 values whose innermost dimension 3
crossVal = tf.cross(a,b)
Provides remainder with respect to each element in the tensor
modVal = tf.mod(a,b)
Provides absolute value of input tensor
negTensor = tf.constant([-2,-4,-6])
absVal = tf.abs(negTensor)
Produces -1 if negative, 0 if value is 0 & 1 if positive value
The input tensor should be either float32, float64, int32, int64
signVal = tf.sign(negTensor, name = "direction")
Provides the reciprocal of a value
The input tensor should be of type float32,float64,int32,complex64,int64
invVal = tf.inv(a, name = "Inverse")
The below function finds the reciprocal of the square root a number; tensor accepts :float32,float64,int32,complex64,int64
Mathematical functions: tf.sqrt(a), tf.round(a): Rounds values to ceiling value; the tensor should be either float or double,
tf.square(a): finds the square
revSqVal = tf.rsqrt(a, name =...
Can you write Python code for it?

import tensorflow as tf
import numpy as np
# The below command creates Op nodes.
# A constant tensor is denoted using a circle.
with tf.name_scope('Values') as scope:
    x = tf.Variable(0, name='x')
    y = tf.constant(5)
    z = x+y
model = tf.initialize_all_variables()

with tf.Session() as session:
    for i in range(5):
        session.run(model)
        print "At Loop: ",i
        data = np.random.randint(10, size=10)
        for j in range(len(data)-1):
                sumVal = tf.add(z,data[0:j+1])
                print "Sum is:",
                result = session.run(sumVal)
                print result

        print "Last sumval: ",result
        with tf.name_scope('Mean') as scope:
                mean = tf.reduce_mean(tf.to_float((sumVal)))
        mean = tf.reduce_mean(tf.to_float((sumVal)))
        print "Reduced Mean is : ",session.run(tf.reduce_mean(tf.to_float((sumVal))))
        _=tf.scalar_summary('Mean', mean)
        merged = tf.merge_all_summaries()
        writer = tf.train.SummaryWriter("/tmp/mnist_logs", session.graph_def)
        model = tf.initialize_all_variables()
        session.run(model)


I am working on the file "Raw/Mitha/INTRODUCTION TO COMPUTING WITH TENSORFLOW.ipynb".
The first task is:
INTRODUCING PLACEHOLDERS
They are used to provide input whenever tensor flow is made to run computation. To mention a dimension of any length, None can be specified in the shape argument. The below example which deals with matrix multiplication of tensors-generated using the random function, provides an understanding of working with placeholder values.
Example 6:
Can you write Python code for it?


I am working on the file "Raw/Mitha/INTRODUCTION TO COMPUTING WITH TENSORFLOW.ipynb".
The first task is:
Tensor Board provides a visualization of the model that you create. It deals with 2 types of connections: Control dependency(denoted using dotted lines) and Data dependency(denoted using solid lines).
To record the variations produced by a particular function, provide them as inputs to scalar_summary ops with a tag name. Histogram_summary provides the distribution of an output variable from a layer.
tf.merge_all_summaries will merge the summaries created within the program, which is then directed to a summary_writer. Specify a logdir for summary writer to write the events.
To get the tensor board launched as below:
- run the command :
        tensorboard --logdir = /tmp/mnist_logs
- Open a browser and navigate to
        http:<server>:6006

Example 5:
Consider a simple model which calculates a new state using the error obtained from the previous state.The functions add, sub forms a node in the graph with their own definite operations. They produce zero or more tensors as inputs for the next set of nodes:
Can you write Python code for it?


I am working on the file "Raw/Mitha/INTRODUCTION TO COMPUTING WITH TENSORFLOW.ipynb".
The first task is:
Tensor dimensionality is denoted using shape, rank and dimension number. Tensor with rank 0 is a scalar. A rank 1 tensor is a vector. A rank 2 tensor is a matrix, ie you can access element using 2 variables like tensor[i,j]; here the dimension is 2D. In the case of rank 3, access it using tensor[i,j,k];here dimension is 3D.
The tensor elements can be subjected to slice, tile operations. The below example deals with slice command.
Example 2:
Can you write Python code for it?


I am working on the file "Rayleigh_damping.ipynb".
The first task is:
Back to top
Rayleigh damping
When Rayleigh damping is considered it means that the damping coefficient is a combination of the two previous ones, that is, it is a multiple of mass and stifnness:
\begin{equation}
\mathbf{C} = \alpha \cdot \mathbf{M} + \beta \cdot \mathbf{K}
\end{equation}
where $\alpha$ and $\beta$ have the previous meanings. In these circunstances, the dynamic equilibrium equation can be written as:
\begin{equation}
\mathbf{M} \times \mathbf{a(t)} + \left[ \alpha \cdot \mathbf{M} + \beta \cdot \mathbf{K} \right] \times \mathbf{v(t)} + \mathbf{K} \times \mathbf{d(t)} = \mathbf{F(t)}
\end{equation}
Proceeding the same as above, one obtains the $NDOF$ independent modal equilibrium equations:
\begin{equation}
\mathbf{M_n} \times \mathbf{\ddot q_n(t)} + \left[ \alpha \cdot \mathbf{M_n} + \beta \cdot \mathbf{K_n} \right] \times \mathbf{\dot q_n(t)} + \mathbf{K_n} \times \mathbf{q_n(t)} = \mathbf{F_n(t)}
\end{equation}
or, equivalently:
\begin{equation}
\mathbf{\ddot q_n(t)} + \left[ \alpha + \beta \cdot \mathbf{\omega_n^2} \right] \cdot \mathbf{\dot q_n(t)} + \mathbf{\omega_n^2} \cdot \mathbf{q_n(t)} = \mathbf{a_n(t)}
\end{equation}
Comparing expressions, one obtains
\begin{equation}
\alpha + \beta \cdot \omega_n^2 = 2 \cdot \zeta \cdot \omega_n \Leftrightarrow \zeta_n = \frac{\alpha}{2 \cdot \omega_n} + \frac{\omega_n \cdot \beta}{2}
\end{equation}
from where it can be seen that the Rayleigh damping is the sum of the previous linear and hyperbolic functions of the vibration frequency $\omega_n$.
Can you write Python code for it?

plt.hold(True)
plt.plot(wn, zn_a+zn_b, label='Rayleigh damping')
plt.plot(wn, zn_a, label='mass proportional')
plt.plot(wn, zn_b, label='stiffness proportional')
plt.xlabel('Vibration frequency $\omega_n$ [rad/s]')
plt.ylabel('Damping coefficient $\zeta_n$ [-]')
plt.legend(loc='best')
plt.grid(True)
plt.xlim([0, 35.])
plt.ylim([0, 0.2])
plt.show()


I am working on the file "Rayleigh_damping.ipynb".
The first task is:
Back to top
Stiffness proportional damping
Stiffness proportional damping means that damping matrix is somehow a multiple of the stiffness matrix:
\begin{equation}
\mathbf{C} = \beta \cdot \mathbf{K}
\end{equation}
where $\beta$ is the constant of stiffness proportionality. In these circunstances, the dynamic equilibrium equation can be written as:
\begin{equation}
\mathbf{M} \times \mathbf{a(t)} + \beta \cdot \mathbf{K} \times \mathbf{v(t)} + \mathbf{K} \times \mathbf{d(t)} = \mathbf{F(t)}
\end{equation}
Proceeding the same as above, one obtains the $NDOF$ independent modal equilibrium equations:
\begin{equation}
\mathbf{M_n} \times \mathbf{\ddot q_n(t)} + \beta \cdot \mathbf{K_n} \times \mathbf{\dot q_n(t)} + \mathbf{K_n} \times \mathbf{q_n(t)} = \mathbf{F_n(t)}
\end{equation}
or, equivalently:
\begin{equation}
\mathbf{\ddot q_n(t)} + \beta \cdot \mathbf{\omega_n^2} \cdot \mathbf{\dot q_n(t)} + \mathbf{\omega_n^2} \cdot \mathbf{q_n(t)} = \mathbf{a_n(t)}
\end{equation}
Comparing expressions, one obtains
\begin{equation}
\beta \cdot \omega_n^2 = 2 \cdot \zeta \cdot \omega_n \Leftrightarrow \zeta_n = \frac{\omega_n \cdot \beta}{2}
\end{equation}
from where it can be seen that the stiffness proportional damping is a linear function of the vibration frequency $\omega_n$.
Can you write Python code for it?

beta = 0.005
zn_b = (beta*wn)/2.
plt.plot(wn, zn_b, label='stiffness proportional')
plt.xlabel('Vibration frequency $\omega_n$ [rad/s]')
plt.ylabel('Damping coefficient $\zeta_n$ [-]')
plt.legend(loc='best')
plt.grid(True)
plt.xlim([0, 35.])
plt.ylim([0, 0.2])
plt.show()


I am working on the file "Rayleigh_damping.ipynb".
The first task is:
We can check that the Rayleigh damping assumes the required values at the desired frequencies, although may vary considerably for other frequencies:
Can you write Python code for it?

zn_a = alpha/(2.*wn)
zn_b = (beta*wn)/2.
plt.hold(True)
plt.plot(wn, zn_a+zn_b, label='Rayleigh damping')
plt.plot(wn, zn_a, label='mass proportional')
plt.plot(wn, zn_b, label='stiffness proportional')
plt.plot(w1, z1, 'o')
plt.plot(w2, z2, 'o')
plt.axvline(w1, ls=':')
plt.axhline(z1, ls=':')
plt.axvline(w2, ls=':')
plt.axhline(z2, ls=':')
plt.xlabel('Vibration frequency $\omega_n$ [rad/s]')
plt.ylabel('Damping coefficient $\zeta_n$ [-]')
plt.legend(loc='best')
plt.xlim([0, 35.])
plt.ylim([0, 0.2])
plt.show()


I am working on the file "Rayleigh_damping.ipynb".
The first task is:
We will produce some plots based on a frequency range to illustrate the concepts:
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

ff = np.linspace(0.01, 6., num=600)
wn = 2.*np.pi*ff


I am working on the file "ReAct/Python/ReAct_Notebook.ipynb".
The first task is:
Explore how the parameters (initial concentrations and kynetic constants) affect the outcome of the response in the cell below. Try to find a link with the explained role of kinase cascades.
Can you write Python code for it?

import numpy as np
from Gilles import *
import matplotlib.pyplot as plt

# Initial conditions
user_input = ['Rec', 10,
              '1M3', 10,
              '1M3P', 0,
              '1M2', 20,
              '1M2P', 0,
              '1M1', 30,
              '1M1P', 0]

# Constants (this is not necessary, they could be filled up already in the reaction tuple)
k = (2,0.05,1,0.5,1,0.5,1)

# Reaction template ((stoch_1,reactant_1,stoch_2,reactant_2),(stoch_1,product_1,stoch_2,product_2),k)
reactions = (
    (1,'Rec'),(),k[0],
    (-1,'Rec',1,'1M3'),(1,'1M3P'),k[1],
    (1,'1M3P'),(1,'1M3'),k[2],
    (-1,'1M3P',1,'1M2'),(1,'1M2P'),k[3],
    (1,'1M2P'),(1,'1M2'),k[4],
    (-1, '1M2P', 1, '1M1'), (1, '1M1P'), k[5],
    (1, '1M1P'), (1, '1M1'), k[6],
)
# dt is used for the deterministic calculation, and the
dt=0.00001
t = np.arange(0, 10, dt)

(solution,(tgill, valsgill, _, _),rows,mode)=ReAct(user_input,reactions,t)

Gillesplot(solution,t,tgill, valsgill,rows,mode)
plt.figure()
Gillesplot(solution,t,tgill, valsgill,rows,mode,['Rec','1M3P','1M2P','1M1P'])
plt.show()


I am working on the file "ReAct/Python/ReAct_Notebook.ipynb".
The first task is:
Gilles.py is the file that contains the important functions, we will go through it to understand the main differences between the deterministic and stochastic solution, but first let's see some examples!
Can you write Python code for it?


I am working on the file "ReAct/Python/ReAct_Notebook.ipynb".
The first task is:
You can copy the content of the file into a new cell, and change the values, explore how the parameters affect the outcome using the cell below.
Can you write Python code for it?

# Initial conditions
user_input = ['A', 100,
              'B', 0]
# Constants (this is not necessary, they could be filled up already in the reaction tuple)
k = (12,8)

# Reaction template ((stoch_1,reactant_1,stoch_2,reactant_2),(stoch_1,product_1,stoch_2,product_2),k)
reactions = (
    (1,'A'),(1,'B'),k[0],
    (1,'B'),(1,'A'),k[1],
)
# dt is used for the deterministic calculation, and the
dt=0.0001
t = np.arange(0, 4, dt)

(solution,(tgill, valsgill, _, _),rows,mode)=ReAct(user_input,reactions,t)

Gillesplot(solution,t,tgill, valsgill,rows,mode)

plt.show()


I am working on the file "Read AFNI.ipynb".
The first task is:
Plot a time series of one voxel.
Can you write Python code for it?

ts = dat[50,50,50,:]
plot(ts)
xlabel("EPI Volume")
ylabel("Signal")

from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

def plotslices(x,y,z,t):

    f, axarr = plt.subplots(1, 3, figsize=(15,6.5))

    axarr[0].imshow(dat[x,:,:,t].T,origin='lower')
    axarr[0].set_title("x=%i"%x)
    axarr[1].imshow(dat[:,y,:,t].T,origin='lower')
    axarr[1].set_title("y=%i"%y)
    axarr[2].imshow(dat[:,:,z,t].T,origin='lower')
    axarr[2].set_title("z=%i"%z)

interact_manual(plotslices,
                x=widgets.IntSlider(min=0,max=dat.shape[0]-1,step=1,value=50),
                y=widgets.IntSlider(min=0,max=dat.shape[1]-1,step=1,value=50),
                z=widgets.IntSlider(min=0,max=dat.shape[2]-1,step=1,value=50),
                t=widgets.IntSlider(min=0,max=dat.shape[3]-1,step=1,value=0));


I am working on the file "Read AFNI.ipynb".
The first task is:
Show one slice from an EPI image.
Can you write Python code for it?

#header["BRICK_TYPES"]
fname = "pb05.s04.pre.r01.blur+tlrc"
hdr,dat = afni.read(fname)

f, axarr = plt.subplots(1, 3, figsize=(12,5.5))
tr = 10
axarr[0].imshow(dat[50,:,:,tr].T,origin='lower')
axarr[0].set_title("x=50")
axarr[1].imshow(dat[:,50,:,tr].T,origin='lower')
axarr[1].set_title("y=50")
axarr[2].imshow(dat[:,:,50,tr].T,origin='lower')
axarr[2].set_title("z=50")


I am working on the file "Read MCNP Output.ipynb".
The first task is:

Calculate number of photons that reach the CCD

Can you write Python code for it?

scatteringlength = 2.84*10**-6 #photon scattering length in Gd2O2S in m
conversionefficiency = 0.16 #electron energy to light energy conversion efficiency
emissionwavelength = 545*10**-9 #lanex emission wavelength in m
emissionenergy = hbar*2*pi*c/emissionwavelength
photonNumberArray = []
for i in range(len(KEsim)):
    photonNumber = 0
    for j in range(len(deptharray[i])):
        photonNumber += conversionefficiency*Eabsorbed_Segments[i][j]*10**6*q/emissionenergy\
        *(j+0.5)/Nsegments # Nabs = Nexc*(Distance from top of lanex)/(Phosphor thickness)
        #where Distance from top of lanex = (SegmentNumber+0.5)/(Nsegments)*(Phosphor thickness)

    photonNumberArray.append(photonNumber)


I am working on the file "Read MCNP Output.ipynb".
The first task is:

Plot CCD signal due to photons and electrons

Can you write Python code for it?

sns.set(context='poster',font_scale=1.5)
sns.set_style("darkgrid")
sns.set_palette(palette='deep')
sns.set_color_codes(palette='deep')

Qph = 0.4 #Quantum efficiency for 545 nm photons
Energy_eh_pair = 3.65 #Energy needed to generate electron-hole pair in Si in eV
Total_Signal = np.multiply(photonNumberArray,Qph) + np.multiply(EelAbsorbedCCD,10**6/Energy_eh_pair) + \
                np.multiply(EphAbsorbedCCD,10**6/Energy_eh_pair)
Total_Relative_Signal = Total_Signal/np.max(Total_Signal)
KEarray = np.multiply(KEsim,1.0)
EspecCalibration = np.vstack((KEarray,Total_Relative_Signal))

plt.subplots(figsize=(8,6))

plt.plot(np.multiply(KEsim,1.0),np.multiply(photonNumberArray,Qph),linewidth=2,color='b',label='Photons')
plt.plot(np.multiply(KEsim,1.0),np.multiply(EelAbsorbedCCD,10**6/Energy_eh_pair),linewidth=2,color='r',label='Electrons')
#plt.plot(np.multiply(KEsim,10**3),np.multiply(EphAbsorbedCCD,10**6/Energy_eh_pair),linewidth=2,color='g',label='X-rays')
plt.plot(EspecCalibration[0],np.multiply(np.max(Total_Signal),EspecCalibration[1]),linewidth=2,color='k',label='Total')
#plt.plot(np.multiply(KEsim,1.0),np.multiply(EelAbsorbedCCD_No_Lanex,10**6/Energy_eh_pair),linewidth=2,color='g',label='No Lanex')

plt.xlabel('Electron Energy (MeV)')
plt.ylabel('Number of Electron-Hole Pairs')
plt.xlim(0,3.0)
plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))
plt.title('Mean CCD signal from\nsingle incident electron')
#plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
plt.legend(loc=4)
plt.tight_layout()


I am working on the file "Read MCNP Output.ipynb".
The first task is:

Plot energy absorbed by each segment of the phoshpor layer for various energies

Can you write Python code for it?


I am working on the file "Read MCNP Output.ipynb".
The first task is:

Plot number of photons absorbed by the CCD

Can you write Python code for it?

plt.subplots(figsize=(12,6))
plt.plot(np.multiply(KEsim,10**3),np.multiply(Eabsorbed,conversionefficiency*10**6*q/emissionenergy/2),linewidth=3,color='r',label='Without Diffusion')
plt.plot(np.multiply(KEsim,10**3),photonNumberArray,linewidth=3,color='b',label='With Diffusion')
plt.xlabel('Electron Kinetic Energy (keV)')
plt.ylabel('Number of Photons')
plt.xlim(150,3000)
plt.title('Number of photons absorbed by CCD due to single electron',y=1.03)
plt.legend(loc=1)
plt.tight_layout()


I am working on the file "Read MCNP Output.ipynb".
The first task is:

Plot total energy absorbed in the phosphor layer

Can you write Python code for it?

plt.subplots(figsize=(12,6))
plt.plot(np.multiply(KEsim,10**3),np.multiply(Eabsorbed,10**3),linewidth=3)
plt.xlim(150,3000)
plt.ylim(0)
plt.xlabel('Kinetic Energy (keV)')
plt.ylabel('Energy Absorbed by Active Layer (keV)')
plt.tight_layout()


I am working on the file "Read MCNP Output.ipynb".
The first task is:

Plot total relative CCD signal due to visible photons, electrons, and x-rays

Can you write Python code for it?

sns.set(context='poster',font_scale=1.5)
sns.set_style("darkgrid")
sns.set_palette(palette='deep')
sns.set_color_codes(palette='deep')

Total_Signal = np.multiply(photonNumberArray,Qph) + np.multiply(EelAbsorbedCCD,10**6/Energy_eh_pair) + \
np.multiply(EphAbsorbedCCD,10**6/Energy_eh_pair)

plt.subplots(figsize=(12,6))
#plt.plot(np.multiply(KEsim,10**3),Total_Signal/np.max(Total_Signal),linewidth=3,color='k')
plt.plot(np.multiply(KEsim,10**3),Total_Signal,linewidth=3,color='k')
plt.xlabel('Electron Kinetic Energy (keV)')
plt.ylabel('CCD Signal')
plt.title('Total Relative CCD Signal')
plt.xlim(0,3000)
plt.tight_layout()


I am working on the file "Read MCNP Output.ipynb".
The first task is:

Read MCNP output files and add the energy deposited in each segment of the phosphor layer into an array along with its associated error

Can you write Python code for it?

deptharray= []
Eabsorbed_Segments = []
Eabsorbed_Segments_Error = []
for i in range(len(KEsim)):
    deptharray.append([])
    Eabsorbed_Segments.append([])
    Eabsorbed_Segments_Error.append([])

    #Directory of the input files
    directory = MCNP_Directory + '/MCNP_Decks/Output/2136Gauss_Oblique_Incidence/Out_{KE}MeV_{Theta}Degrees'\
    .format(KE=str(round(KEsim[i],3)),Theta=str(round(np.arccos(CosThetaSim[i])*360/(2*pi),1)))

    for segmentnumber in range(Nsegments-1):
        segment = segmentnumber+100 #segment number
        deptharray[i].append(segmentnumber*phosphorthickness/Nsegments) #depth for each segment in microns
        printflag = False
        with open(directory) as searchfile:
            for line in searchfile:
                left,sep,right = line.partition('       -{segmentlabel}  '.format(segmentlabel=str(segment)))
                if printflag:
                    Eabs_per_g = float(line[17:28])
                    Eabs_per_g_Error = float(line[29:35])
                    Eabsorbed_Segments[i].append(Eabs_per_g*segmentmass)
                    Eabsorbed_Segments_Error[i].append(Eabs_per_g_Error*segmentmass)
                    printflag = False
                if sep:
                    printflag = True


I am working on the file "Read MCNP Output.ipynb".
The first task is:

Read MCNP output files and add the energy deposited in the photoactive CCD layer into an array along with its associated error

Can you write Python code for it?

EphAbsorbedCCD = [] #x-rays
EphAbsorbedCCDError = []
EelAbsorbedCCD = [] #electrons
EelAbsorbedCCDError = []
CCDmass = 2.09610*10**-4 #mass of photoactive layer of CCD
for i in range(len(KEsim)):
    #Directory of the input files
    directory = MCNP_Directory + '/MCNP_Decks/Output/2136Gauss_Oblique_Incidence/Out_{KE}MeV_{Theta}Degrees'\
    .format(KE=str(round(KEsim[i],3)),Theta=str(round(np.arccos(CosThetaSim[i])*360/(2*pi),1)))

    printflag = False
    with open(directory) as searchfile:
        firstoccurence = False
        for line in searchfile:
            left,sep,right = line.partition('cell  6')
            if printflag:
                Eabs_per_g = float(line[17:28])
                Eabs_per_g_Error = float(line[29:35])
                if firstoccurence:
                    EphAbsorbedCCD.append(Eabs_per_g*CCDmass)
                    EphAbsorbedCCDError.append(Eabs_per_g_Error*CCDmass)
                else:
                    EelAbsorbedCCD.append(Eabs_per_g*CCDmass)
                    EelAbsorbedCCDError.append(Eabs_per_g_Error*CCDmass)
                printflag = False
            if sep: # True iff 'cell  6' in line
                printflag = True
                firstoccurence = not firstoccurence


I am working on the file "Read MCNP Output.ipynb".
The first task is:
No lanex case
Can you write Python code for it?

# EphAbsorbedCCD_No_Lanex = [] #x-rays
# EphAbsorbedCCDError_No_Lanex = []
# EelAbsorbedCCD_No_Lanex = [] #electrons
# EelAbsorbedCCDError_No_Lanex = []
# CCDmass = 2.09610*10**-4 #mass of photoactive layer of CCD
# for i in range(len(KEsim)):
#     #Directory of the input files
#     directory = '/home/drake/Desktop/Physics/Research/Python/MCNP_Code/MCNP_Decks/Output/1.5um_CCD_No_Lanex_Output/Out_{KE}MeV_{Theta}Degrees'\
#     .format(KE=str(round(KEsim[i],3)),Theta=str(round(np.arccos(CosThetaSim[i])*360/(2*pi),1)))

#     printflag = False
#     with open(directory) as searchfile:
#         firstoccurence = False
#         for line in searchfile:
#             left,sep,right = line.partition('cell  2')
#             if printflag:
#                 Eabs_per_g = float(line[17:28])
#                 Eabs_per_g_Error = float(line[29:35])
#                 if firstoccurence:
#                     EphAbsorbedCCD_No_Lanex.append(Eabs_per_g*CCDmass)
#                     EphAbsorbedCCDError_No_Lanex.append(Eabs_per_g_Error*CCDmass)
#                 else:
#                     EelAbsorbedCCD_No_Lanex.append(Eabs_per_g*CCDmass)
#                     EelAbsorbedCCDError_No_Lanex.append(Eabs_per_g_Error*CCDmass)
#                 printflag = False
#             if sep: # True iff 'cell  6' in line
#                 printflag = True
#                 firstoccurence = not firstoccurence


# plt.subplots(figsize=(12,6))
# plt.plot(np.multiply(KEsim,10**3),np.multiply(EelAbsorbedCCD_No_Lanex,10**3),linewidth=3,color='r')
# plt.plot(np.multiply(KEsim,10**3),np.multiply(EphAbsorbedCCD_No_Lanex,10**3),linewidth=3,color='g')
# plt.xlabel('Kinetic Energy (keV)')
# plt.ylabel('Energy (keV)')
# plt.title('Energy deposited in photoactive layer of CCD due to single electron')
# plt.subplots_adjust(left=0.12,bottom=0.14) #Adjust spacing to prevent clipping of x and y labels
# plt.show()


I am working on the file "Read MCNP Output.ipynb".
The first task is:
Write to csv
Can you write Python code for it?

Total_Relative_Signal = Total_Signal/np.max(Total_Signal)
KEarray = np.multiply(KEsim,10**3)
EspecCalibration = np.vstack((KEarray,Total_Relative_Signal))

#np.savetxt('Total_Relative_CCD_Signal_vs_Energy_(keV)_Normal_Incidence.csv',np.transpose(EspecCalibration),delimiter=',')


I am working on the file "ReadNboDip.ipynb".
The first task is:
NLMO part of the DIPOLE MOMENT ANALYSIS: section is now shown in Table 1. The corresponding *_dip.csv file was saved with the path shown in the previous cell.
Can you write Python code for it?

# Print html formatted table from the loaded css file
HTML(df.to_html(classes = 'grid', escape=False))


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
A single element:
Can you write Python code for it?


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
As suggested by Hans, I should check the second number to be in between 0 and -30, since this is backscatter value. First part looks like a date sequence... OK!
Extract data from a gridpoint
So considering the email, following steps are undertaken:
* Decide about a number/grid point to select
* Check for which rows this number is in the file and save these indices
* Select the rows from the .dat file based on the saved indices
Let's try it out, based on the idx_data and the data arrays:
Can you write Python code for it?

# Decide about a grid point
grid_point = 2301829

# select rows/position for the given grid point
indxs_point = np.where(idx_data == grid_point)

# Get these rows out of the data-array
my_selection = data[indxs_point]

my_selection


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
Extracting all grid points, all in separate files
Can you write Python code for it?

all_data = convert_ascat_to_matlab("./ASCAT/1323",
                                   grid_point_id='all',
                                   byte2skip=208)


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
Hans works in Matlab, so I'd like to provide the necessary data as a .mat-file. Exporting to .mat files is available as a scipy package extension:
Can you write Python code for it?


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
I read a list of numbers, as described in the email. So the grid-cells are clustered together for a set of lines. Let's see how the unique values look like:
Can you write Python code for it?


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
INDEX FILE
The data-type structure was provided (and apparently numpy-styling):
Can you write Python code for it?


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
Received information about dataset (email-conversation)
Provided files in a folder (ftp)
Can you write Python code for it?


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
So, let's test the functionality as a function...
Extracting a single grid point, by naming the ID
Can you write Python code for it?

my_data = convert_ascat_to_matlab("./ASCAT/1323", grid_point_id=2301795, byte2skip=208)


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
Some testing on how to export it: different grid points in a single file is possible, but maybe not feasible for >1000 grid_points...
Can you write Python code for it?


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
This data sets contains 1379 different grid points that can be selected
DAT-FILE itself
Reading in, considering the provided data-type and the 208 bit header to skip
Can you write Python code for it?

struct2 = np.dtype([('jd', np.double),  ('sig', np.float32), ('sig_noise', np.float32),
                   ('dir', np.dtype('S1')),    ('pdb',np.ubyte), ('azcorr_flag',
                                                                  np.dtype([('f', np.ubyte),
                                                                            ('m',np.ubyte),
                                                                            ('a', np.ubyte)]))])

f = open(''.join(["./ASCAT/", filename, ".dat"]), "rb")  # reopen the file
f.seek(208, os.SEEK_SET)  # seek

data = np.fromfile(f, dtype=struct2)

data


I am working on the file "Read_ASCAT_data.ipynb".
The first task is:
We have to skip the  first 208 bytes to avoid the reading of the header, so I checked on Stackoverflow how to do this: http://stackoverflow.com/questions/14245094/how-to-read-part-of-binary-file-with-numpy
Can you write Python code for it?

f = open("".join(["./ASCAT/", filename, ".idx"]), "rb")  # reopen the file
f.seek(208, os.SEEK_SET)  # seek

idx_data = np.fromfile(f, dtype=struct1).astype('int32') # astype erbij om de omzetting te verzekeren


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
1. Python modules
We import modules to read the netCDF file and to perform basic operations on the data.
Can you write Python code for it?

import cf
from netCDF4 import Dataset
import numpy as np


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
2. Read data
As most of the file content is compliant with the CF conventions, we can use the cf package to read the variable we need:
* Longitude and latitude
* Time
* Depth
* Salinity
2.1 Variable and coordinates
Can you write Python code for it?

f = cf.read(datafile)
salinity = f.select('sea_water_salinity')[0].array
ftime = f.select('time')[0].array
lon = f.select('lon')[0].array
lat = f.select('lat')[0].array
depth = f.select('depth')[0].array
f.close()


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
2.2 Select one depth level
We can take the data on a given depth level to perform basic operations and a plot. Let's try with a depth of 100 meters.
Can you write Python code for it?


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
2.3 Quality flags
A quality flag (QF) is attributed to each measurement. We can easily count the measurements for each QF, using a loop from 0 to 9.
Can you write Python code for it?

for qf in np.arange(0, 10):
        print 'QF=%s: %s profiles' % (qf, np.sum(salinity_atdepth_QC == qf))


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
4. Plotting
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib as mpl
from matplotlib import colors


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
4.2 Colored histogram
We start by defining the colormap and the limits for the histogram.
Can you write Python code for it?

cmap = plt.cm.get_cmap('RdYlBu_r')
Y, X = np.histogram(salinity_atdepth_masked, np.arange(32., 40., 0.2))


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
4.3 Scatter plot
We keep the same colormap as for the histogram and we define the range of values for the scatter plot.
Can you write Python code for it?

norm = colors.Normalize(vmin=32., vmax=39.)


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
Now we take the salinity and its corresponding flags at that level.
Can you write Python code for it?

salinity_atdepth = salinity[:, depthindex]
salinity_atdepth_QC = salinityQC[:, depthindex]


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
Then we make the plot.
Can you write Python code for it?

x_span = X.max() - X.min()
C = [cmap(((x - X.min()) / x_span)) for x in X]
fig = plt.figure(figsize=(8, 6))
plt.bar(X[:-1], Y, color=C, width=X[1]-X[0])
plt.xlabel('Salinity')
plt.ylabel('Number of data points')
plt.show()


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
This would mean that we have 26693 measurements available at that depth. However, some values can be masked, because the profile doesn't reach that depth. So to compute the real number of observations, we can use count which is an in-place method of cf.
Can you write Python code for it?

goodmeasurements = salinity_atdepth.count()
print str(goodmeasurements)  + ' data points available'


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
We can do better by including coastline, scale and other labels.
4.4 Scatter on a map
We first need to import the Basemap module to plot data on a map, then the projection has to be defined. We select a Mollweide projection centered at 0º longitude, with a crude resolution for the coastline.
Can you write Python code for it?

from mpl_toolkits.basemap import Basemap
m = Basemap(projection='moll', lon_0=0, resolution='c')


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
We compute the index of the level corresponding to that depth.
Can you write Python code for it?

depthindex = np.argmin(abs(depth - mydepth))
print 'depthindex = ' + str(depthindex)


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
We create a scatter plot with colors scaled 32 and 39 and indicate the positions of the minimum and maximum salinity (squares).
Can you write Python code for it?

fig = plt.figure(figsize=(10,8))
scat = plt.scatter(lon, lat, s=10, c=salinity_atdepth_masked, edgecolor='None', cmap=cmap, norm=norm)
plt.plot(lonmax, latmax, 'rs', ms=10)
plt.plot(lonmin, latmin, 'bs', ms=10)
plt.colorbar(scat, extend='both')
plt.show()


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
We set the fontsize to 20 pts.
Can you write Python code for it?


I am working on the file "Read_CORA_dataset.ipynb".
The first task is:
We will only consider the observations with QF=1. To do this, we could try selecting only the values of the array that satisfy the condition:
Can you write Python code for it?

salinity_plot = salinity_atdepth[salinity_atdepth_QC == 1]
print salinity_plot.shape
print lon.shape


I am working on the file "Reading papers from user.ipynb".
The first task is:
Count the references and number for each author
Can you write Python code for it?


I am working on the file "Reading papers from user.ipynb".
The first task is:
Example, to obtain my h-index
Can you write Python code for it?

author_info = scopus_get_author(MY_AUTHOR_ID)
print_json(author_info)
h_index = author_info['h-index']
print("My automatic h_index is {}".format(h_index))


I am working on the file "Reading papers from user.ipynb".
The first task is:
Get authors from a list
Can you write Python code for it?

def get_author_info(paper_id):
    url = ("abstract/scopus_id/"
          + paper_id
          + "?field=authors,title,publicationName,volume,issueIdentifier,"
          + "prism:pageRange,coverDate,article-number,doi,citedby-count,prism:aggregationType")

    resp = scopus_get_info_api(url, json=True)
    results = resp['abstracts-retrieval-response']
    authors_info = results['authors']['author']
    authors_id = [au['ce:indexed-name'] for au in authors_info]
    return authors_id

from collections import defaultdict

def get_authors_list(papers_id):
    number = defaultdict(int)

    for i, paper_id in enumerate(papers_id):
        authors = get_author_info(paper_id)

        for author in authors:
            number[author] += 1

    return number

authors = get_authors_list(df['id'])
print(authors)

def show_author_list(authors):
    names = authors.keys()
    names = sorted(names, key=lambda k: authors[k], reverse=True)

    for name in names:
        print("{}: {}".format(name, authors[name]))

show_author_list(authors = get_authors_list(df.id))


I am working on the file "Reading papers from user.ipynb".
The first task is:
Get complete reference of a paper
Can you write Python code for it?

def get_scopus_info(SCOPUS_ID):
    url = ("abstract/scopus_id/"
          + SCOPUS_ID
          + "?field=authors,title,publicationName,volume,issueIdentifier,"
          + "prism:pageRange,coverDate,article-number,doi,citedby-count,prism:aggregationType")

    resp = scopus_get_info_api(url, json=True)
    results = resp['abstracts-retrieval-response']
    authors_info = results['authors']
    info = results['coredata']

    fstring = '{authors}, {title}, {journal}, {volume}, {articlenum}, ({date}). {doi} (cited {cites} times).\n'
    return fstring.format(authors=', '.join([au['ce:indexed-name'] for au in authors_info['author']]),
                          title=info['dc:title'],
                          journal=info['prism:publicationName'],
                          volume=info.get('prism:volume') or 1,
                          articlenum=info.get('prism:pageRange') or
                            info.get('article-number'),
                          date=info['prism:coverDate'],
                          doi='doi:' +(info.get('prism:doi') or 'NA'),
                        cites=int(info['citedby-count']))

df_lasts = df[df['year']=='2015']

for id in df.sort(['citations'], ascending=[0])['id']:
    #print("id: '{}'".format(id))
    print(get_scopus_info(id))


I am working on the file "Reading papers from user.ipynb".
The first task is:
Get the number of journals with the results
Can you write Python code for it?

lsgo_journal = papers_lsgo.groupby(['journal']).sum()
lsgo_journal.columns = ['number']
lsgo_journal = lsgo_journal.sort('number', ascending=False)
lsgo_journal = lsgo_journal[lsgo_journal['number']>0]

print(lsgo_journal)

papers_lsgo = papers_lsgo.sort('citations', ascending=False)
papers_lsgo = papers_lsgo[papers_lsgo['citations']>0]

for id in papers_lsgo['id'][:10]:
    print(get_scopus_info(id))