I am working on the file "SOF_Execucao_Orcamentaria_SMESP.ipynb".
The first task is:
Atenção: as consultas podem demorar horas, a depender da quantidade de anos requerida; verifique se o número de anos acima é realmente necessário; faça apenas isso uma vez, e guarde a base para análises futuras
Can you write Python code for it?

df_empenhos_lista = []

for ano in anos:

    request_empenhos = requests.get('{base_url}/consultaEmpenhos?anoEmpenho={ano}&mesEmpenho=12&codOrgao=16'.format(base_url=base_url, ano=ano),
                                            headers=headers, verify=True).json()

    number_of_pages = request_empenhos['metadados']['qtdPaginas']

    todos_empenhos = []

    todos_empenhos = todos_empenhos + request_empenhos['lstEmpenhos']

    if number_of_pages>1:

        for p in range(2, number_of_pages+1):

            request_empenhos = requests.get('{base_url}/consultaEmpenhos?anoEmpenho={ano}&mesEmpenho=12&codOrgao=16'.format(base_url=base_url, ano=ano) + pagination.format(PAGE=p),
                                            headers=headers, verify=True).json()

            todos_empenhos = todos_empenhos + request_empenhos['lstEmpenhos']

    df_empenhos = pd.DataFrame(todos_empenhos)

    df_empenhos_lista.append(df_empenhos)

df_empenhos_serie = pd.concat(df_empenhos_lista, ignore_index=True)

df_empenhos_serie.columns


I am working on the file "SOF_Execucao_Orcamentaria_SMESP.ipynb".
The first task is:
Empenhos
Empenho é o ato em que autoridade verifica a existência do crédito orçamentário e autoriza a execução da despesa (por exemplo, para realizar uma licitação). A partir daí, os valores vão sendo liquidados e pagos conforme a execução de um contrato.
A API fornece apenas uma página na consulta. O script abaixo checa a quantidade de páginas nos metadados da consulta e itera o número de vezes necessário para obter todas as páginas.
Opção 1. Um ano inteiro
Vamos ver quanto a Secretaria Municipal de Educação empenhou de seu orçamento em 2017.
Can you write Python code for it?

pagination = '&numPagina={PAGE}'
ano_empenho = 2017

request_empenhos = requests.get('{base_url}/consultaEmpenhos?anoEmpenho={ano}&mesEmpenho=12&codOrgao=16'.format(base_url=base_url, ano=ano_empenho),
                                            headers=headers, verify=True).json()

    number_of_pages = request_empenhos['metadados']['qtdPaginas']

    todos_empenhos = []

    todos_empenhos = todos_empenhos + request_empenhos['lstEmpenhos']

    if number_of_pages>1:

        for p in range(2, number_of_pages+1):

            request_empenhos = requests.get('{base_url}/consultaEmpenhos?anoEmpenho={ano}&mesEmpenho=12&codOrgao=16'.format(base_url=base_url, ano=ano_empenho) + pagination.format(PAGE=p),
                                            headers=headers, verify=True).json()

            todos_empenhos = todos_empenhos + request_empenhos['lstEmpenhos']

df_empenhos = pd.DataFrame(todos_empenhos)


I am working on the file "SOF_Execucao_Orcamentaria_SMESP.ipynb".
The first task is:
Fontes de recursos
Agrupamento dos empenhos por fonte de recursos:
Can you write Python code for it?

fonte = pd.pivot_table(df_empenhos_serie,
                       values=['valLiquidado', 'valPagoExercicio'],
                       index=['txtDescricaoFonteRecurso'],
                       aggfunc=np.sum).sort_values('valPagoExercicio', axis=0, ascending=False, inplace=False, kind='quicksort', na_position='last')

fonte


I am working on the file "SOF_Execucao_Orcamentaria_SMESP.ipynb".
The first task is:
Maiores despesas
Aqui vamos produzir a lista das 15 maiores despesas da Educação neste período:
Can you write Python code for it?

despesas = pd.pivot_table(df_empenhos_serie,
                              values=['valLiquidado', 'valPagoExercicio'],
                              index=['numCpfCnpj', 'txtRazaoSocial', 'txtDescricaoProjetoAtividade'],
                              aggfunc=np.sum).sort_values('valPagoExercicio', axis=0, ascending=False, inplace=False, kind='quicksort', na_position='last')

despesas.head(15)


I am working on the file "SOF_Execucao_Orcamentaria_SMESP.ipynb".
The first task is:
Orçamento
Primeiro, vamos ter uma visão geral do que foi orçado para a Secretaria Municipal de Educação desde 2011 até o ano corrente, bem como os valores congelados e já executados. Isso é possível com a consulta "Despesas"
Can you write Python code for it?

df_lista = []
a = 0

for ano in anos:
    """consulta todos os anos da lista acima"""

    url_orcado = '{base_url}/consultarDespesas?anoDotacao={ano}&mesDotacao=12&codOrgao=16'.format(base_url=base_url, ano=ano)
    request_orcado = requests.get(url_orcado,
                                headers=headers,
                                verify=True).json()

    df_orcado = pd.DataFrame(request_orcado['lstDespesas'])
    df_lista.append(df_orcado)
    df_lista[a].loc[:,'anoExercicio'] = str(ano)
    #adiciona uma coluna com o ano de exercício, já que esta informação não vem na API
    a += 1

df_total = pd.concat(df_lista, axis=0, ignore_index=True)

df_total


I am working on the file "SOF_Execucao_Orcamentaria_SMESP.ipynb".
The first task is:
Passo 4. Quer salvar um csv?
O objetivo deste tutorial não era fazer uma análise exaustiva da base, mas apenas mostrar o que é possível a partir do consumo da API. Você também pode salvar toda a base de empenhos num arquivo .csv e trabalhar no seu Excel (super te entendo). O Pandas também ajuda nisso! Assim:
Can you write Python code for it?


I am working on the file "SPF/develop/get_Individ_Uncert.ipynb".
The first task is:
bins =np.array([-2.0,-1.0,-0.5,0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0,5.0])
orig_hist = np.array(h).astype(float)
norm_hist = orig_hist/float(sum(orig_hist))
mid_points = (bins[1:] + bins[:-1])/2
bin_widths = np.diff(bins)
MeanCrude = np.dot(norm_hist,mid_points)
MeanCrude
VarCrude  = np.dot(norm_hist,(mid_points-MeanCrude)**2)
VarCrude
total = 0
for i in range(0,len(norm_hist)):
    total += norm_hist[i]*mid_points[i]
total
for i in range(0,len(norm_hist)):
    tt[i] = (mid_points[i] - MeanCrude)**2
tt
VarSheppard = VarCrude - (BinWidth**2)/12
VarSheppard
MeanCrude = sum(Prob . CatMid);                    %crude mean and variance
VarCrude  = sum(Prob . (CatMid-MeanCrude).^2);
BinWidth    = mean(diff(CatBounds(:)));
VarSheppard = VarCrude - BinWidth^2/12;       %variance, Sheppard's correction
def Entr(h):
    if not all(h==0):
        bins =np.array([-2.0,-1.0,-0.5,0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0,5.0])
        orig_hist = np.array(h).astype(float)
        norm_hist = orig_hist/float(sum(orig_hist))
        bin_widths = np.diff(bins)
    np.dot(h,np.log(h/bin_widths))
    bin_density = h.astype(float) / np.dot(bin_widths, h)
    popt
else:
    popt = [float('nan'),float('nan')]
return popt[1]
Can you write Python code for it?

mask = df.columns.str.contains(',')
mask

df.columns[mask]


def fit_normal_to_hist(h):
    if not all(h==0):
        bins =np.array([-2.0,-1.0,-0.5,0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0,5.0])
        orig_hist = np.array(h).astype(float)
        norm_hist = orig_hist/float(sum(orig_hist))
        mid_points = (bins[1:] + bins[:-1])/2
        popt,pcov = opt.curve_fit(lambda x,mu,sig: stats.norm.pdf(x,mu,sig), mid_points,norm_hist)
    else:
        popt = [float('nan'),float('nan')]
    return popt[1]

def ZL_std(h):
    intervals =[[-2.0,-1.1],[-1.0,-0.6],[-0.5,-0.1],[0.0,0.4],[0.5,0.9],[1.0,1.4],
           [1.5,1.9],[2.0,2.4],[2.5,2.9],[3.0,3.4],[3.5,3.9],[4.0,5.0]]
    if not all(h==0):
        sum_i1 = 0
        sum_i2 = 0
        for i in range(1,len(h)):
            p = h[i]/100
            v1,v2 = intervals[i]
            sum_i1 += p*(v2**3 - v1**3)/(3*(v2-v1))
            sum_i2 += p*(v2**2 - v1**2)/(2*(v2-v1))
            zl_std = np.sqrt(sum_i1 - sum_i2**2)
    else:
            zl_std = float('nan')
    return zl_std

def Hist_std(h):
    if not all(h==0):
        bins =np.array([-2.0,-1.0,-0.5,0.0,0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0,5.0])
        orig_hist = np.array(h).astype(float)
        norm_hist = orig_hist/float(sum(orig_hist))
        mid_points = (bins[1:] + bins[:-1])/2
        MeanCrude = np.dot(norm_hist,mid_points)
        VarCrude  = np.dot(norm_hist,(mid_points-MeanCrude)**2)
        bin_widths = np.diff(bins)
        BinWidth = bin_widths.mean()
        VarSheppard = VarCrude - (BinWidth**2)/12       #variance, Sheppard's correction
        hist_std = np.sqrt(VarSheppard)
    else:
        hist_std = float('nan')
    return hist_std


I am working on the file "SPINSpy Tutorial.ipynb".
The first task is:
Note that this is ordered as (Nx,Ny,Nz). If we had wanted MATLAB style ordering (Ny,Nx,Nz), we could have used the optional argument ordering = 'matlab' (useful for some plotting). The default is ordering = 'natural' (useful because it feels right).
If we had high resolution or wanted to load multiply variables into memory simultaneously, we might start running into memory issues. For example, let's look at the size of rho.
Can you write Python code for it?


I am working on the file "SPINSpy Tutorial.ipynb".
The first task is:
Now that we have some packages to work with, let's see what files we have. These sample files, provided by (more correctly stolen from) David, and are part of a mode-2 simulation.

rho.15: binary file with the output 15 of the density field
[x,y,z]grid: binary file with the grids
matpy: a package with some useful functions (such as darkjet, just for you Marek)
spinspy: a package with the SPINS readers

If we want to use our shiny python scripts, we'll need to import them too.
Can you write Python code for it?

import spinspy as spy
import matpy as mp


I am working on the file "SPINSpy Tutorial.ipynb".
The first task is:
Perfect, now we have our grid vectors and they are fairly small (a few kb). The readers package also provides a function to let you load in all of the grid vectors in one fell swoop.
x,y,z = spy.grid()
We can also load in grid information, such as the domain size and limits. This is stored in a class that has a method called `display'. These are illustrated below.
Can you write Python code for it?

data = spy.get_shape()
print(data.Nx,data.Ny,data.Nz)
print('---')
data.display()


I am working on the file "SPINSpy Tutorial.ipynb".
The first task is:
Plotting

We now have our grid vectors and rho field loaded into memory, so it's time to try some plotting! A standard plotting library in python is matplotlib, which replicates many of MATLAB's plotting functions. For example, here we have a pcolor and contour plot of an x-z slice of rho.
Can you write Python code for it?

rho = spy.reader('rho', 30, [0,-1], [0,-1], [0,-1])
plt.figure(1)
t0 = time.clock()
plt.contour(x,z,rho[:,192/2,:].transpose());
t1 = time.clock()
print('Time to plot: {0:1.2e}'.format(t1-t0))
plt.title('Mode-2 Waviness');
plt.xlabel('x');
plt.ylabel('z');

rho = spy.reader('rho', 30, [0,-1], [0,-1], [0,-1])
plt.figure(2)
t0 = time.clock()
plt.pcolor(x,z,rho[:,1,:].transpose());
t1 = time.clock()
print('Time to plot: {0:1.2e}'.format(t1-t0))
plt.title('Mode-2 Waviness');
plt.xlabel('x');
plt.ylabel('z');

rho = spy.reader('rho', 30, [0,-1], [0,-1], [0,-1])
plt.figure(3)
t0 = time.clock()
plt.pcolormesh(x,z,rho[:,192/2,:].transpose(), cmap='darkjet')
t1 = time.clock()
print('Time to plot: {0:1.2e}'.format(t1-t0))
plt.title('Mode-2 Waviness')
plt.xlabel('x')
plt.ylabel('z')
plt.axis('tight')
plt.show()
#plt.gca().set_xlim((data.xlim[0],data.xlim[1]))
#plt.gca().set_ylim((data.zlim[0],data.zlim[1]))


I am working on the file "SPINSpy Tutorial.ipynb".
The first task is:
Reading Full Files

Now that we have all of our tools, let's start doing! To begin, let's load our density field.
Can you write Python code for it?

cd /scratch/orca/scratch/kglamb/ddeepwel/asymkh/

ls rho.*

rho = spy.reader('rho', 30, [0,-1], [0,-1], [0,-1])


I am working on the file "SPINSpy Tutorial.ipynb".
The first task is:
What the heck did that just do? Let's break it down and look at the inputs.

'rho' : This tells the function which variable to load (in this case, rho)
15 : This is the index, so we will load rho.15. x,y,z variables done take an index.
[0,-1] : The x-range to load. This is shorthand and is equivalent to ':' from MATLAB.
[0,-1] : The y-range
[0,-1] : The z-range

Here we have loaded the entire thing. Alternatively, you could have given a single value or a list of indices. That is, giving 0 would take the first slice, [0,100] takes the first 100 elements, and [0,2,4,6] takes the first, third, fifth, and, seventh elements.
To load an x-z slice we could have done:
rho = spy.reader('rho',15,[0,-1],0,[0,-1])
Can you write Python code for it?


I am working on the file "SQL Made Simple.ipynb".
The first task is:
In the "with scope" below, we create a new table by calling a method saved inside the context manager itself. The a text file is read in and each line gets saved to a row in the Glossary table.  Again, the context manager has a method for inserting rows.
Can you write Python code for it?

with Glossary(SQL_DB) as dbx:  # <--- dbx returned by __enter__

    # for testing __exit__ in case of an exception
    # raise NoConnection

    dbx.create_table()
    FILE = os.path.join(dbx.db.target_path, "glossary.txt")

    with open(FILE, 'r', encoding='UTF-8') as gloss:
        lines = gloss.readlines()

    for line in lines:
        if len(line.strip()) == 0:
            continue
        term, definition = line.split(":", 1)
        right_now = dbx.db.mod_date()
        dbx.save_term(term[2:].strip(), definition.strip(), right_now, dbx.db.user_initials)


I am working on the file "SQL Made Simple.ipynb".
The first task is:
We now have a resuable context that will connect and disconnect from our database.
Lets use it to query the Glossary table...
Can you write Python code for it?

with Glossary(SQL_DB) as dbx:

    query = "SELECT gl_term, gl_definition FROM Glossary ORDER BY gl_term"

    dbx.db.curs.execute(query)  # gets the data

    print("{:^80}".format("GLOSSARY OF TERMS"))
    print("-" * 80)
    print("Term                                |Abbreviated Definition   " )
    print("-" * 80)

    for term in dbx.db.curs.fetchmany(10):  # fetchone(), fetchmany(n), fetchall()
        print("{:35} | {:45}".format(term[0], term[1][:45]))


I am working on the file "SQL/SQLquerySizeCalculator.ipynb".
The first task is:
this means the SQL query above would take 0.000124 TB to run.
Can you write Python code for it?

query2 = """SELECT value
            FROM `bigquery-public-data.openaq.global_air_quality`
            WHERE country = 'NL'"""
# ! the quotations marks around 'bigquery..._quality' are NOT quotation marks, they are and have to be 'backticks': ` !

open_aq.estimate_query_size(query2)


I am working on the file "SQL_Basics.ipynb".
The first task is:
Create an user
While creating an user, we'll see if to give user some privileges such as being able to select columns out of some tables:
Can you write Python code for it?

GRANT ALL PRIVILEGES ON *.* TO 'username'@'%' IDENTIFIED BY 'userpasswd' WITH GRANT OPTION;


I am working on the file "SQL_Basics.ipynb".
The first task is:
MySQL
Can you write Python code for it?


I am working on the file "SQL_Basics.ipynb".
The first task is:
Now, the tables are created. We are ready to practice. Here are my solutions.
Q1: 男性與女性的會員數量與總點播次數
Can you write Python code for it?

select userprofile.gender as gender,count(distinct(userprofile.userid)) as numMembers,count(userdemand.time) as totalPlayTimes
from userprofile left outer join userdemand
on userprofile.userid=userdemand.userid
group by gender

df=pd.read_csv('q1.csv')

df

sum(df['totalPlayTimes']) # 總點播次數


I am working on the file "SQL_Basics.ipynb".
The first task is:
Q3: 哪一天是會員透過我們的音樂網站聆聽音樂最多的一天？
解一：
Can you write Python code for it?

SELECT substring(time,1,10) as date,count(*) as nPlayTimes
FROM exercises.userdemand
group by date
order by nPlayTimes desc;


I am working on the file "SQL_Basics.ipynb".
The first task is:
Q4: 會員年齡層分布
使用select (case when condition then xxx else xxx end)
解一 (用with as)：
Can you write Python code for it?

with tmp as (SELECT userid,age,(case
when age>=0 and age<20 then '0-19'
when age>=20 and age<40 then '20-39'
when age>=40 and age<65 then '40-64'
when age>=65 then '65-'
end) as ageZone
FROM exercises.userprofile)
select ageZone,count(*) as numMembers from tmp group by ageZone


I am working on the file "SQL_Basics.ipynb".
The first task is:
https://www.techonthenet.com/oracle/grant_revoke.php

with grant option: meaning that this user is able to grant privileges for some other people.
*.* means the user can access to anyDB.anyTable.
@'%' means that the user is came from any IP.

Create DB & tables:
commands:

create table mytable

load data infile filepath into table tablename

https://www.tutorialspoint.com/mysql/mysql-data-types.htm
Can you write Python code for it?

CREATE DATABASE exercises

CREATE TABLE userprofile (
    userid VARCHAR(20) NOT NULL,
    gender char(1),
    age int,
    country VARCHAR(20),
    registered VARCHAR(20),
    PRIMARY KEY (userid)
);

LOAD DATA INFILE '/users/raemoen/BC3/userprofile.tsv'
INTO TABLE userprofile
FIELDS TERMINATED BY '\t'
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
--IGNORE 1 ROWS;

CREATE TABLE userdemand (
    userid VARCHAR(20) NOT NULL,
    time VARCHAR(50),
    artname VARCHAR(100),
    traname VARCHAR(100),
    FOREIGN KEY(userid) REFERENCES userprofile(userid)
);

LOAD DATA INFILE '/users/raemoen/BC3/userdemand.tsv'
INTO TABLE userdemand
FIELDS TERMINATED BY '\t'
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
--IGNORE 1 ROWS;

CREATE TABLE calendar (
    yMd VARCHAR(100),
    dofw VARCHAR(100),
    wnofy VARCHAR(100)
);

LOAD DATA INFILE '/users/raemoen/BC3/calendar.csv'
INTO TABLE calendar
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n';
#IGNORE 1 ROWS;


I am working on the file "SQL_Basics.ipynb".
The first task is:
正確。各國家點播次數相加為總點播次數。
亦可用mysql指令驗證，如下：
Can you write Python code for it?

with temp as (select userprofile.country as country,count(distinct(userprofile.userid)) as numMembers,count(userdemand.time) as totalPlayTimes
from userprofile left outer join userdemand
on userprofile.userid=userdemand.userid
group by country
order by numMembers desc
)
select sum(totalPlayTimes) from temp;


I am working on the file "SQL_Basics.ipynb".
The first task is:
解二 (用subquery)：
Can you write Python code for it?

select  t.ages,count(t.age)
from (
select age,
 CASE
  WHEN age between 0 and 19 THEN "0-19"
  WHEN age between 20 and 39 THEN "20-39"
  WHEN age between 40 and 64 THEN "40-64"
  else "65--"
END ages
FROM exercises.userprofile) t
group by t.ages;

df=pd.read_csv('q4.csv')

df


I am working on the file "SQL_Basics.ipynb".
The first task is:
解二(此法不好，因建暫存table是不必要的)：
Can you write Python code for it?

with temp as
(select userid,substring(time,1,10) as time
from userdemand
)
select time,count(*) as nPlayTimes from temp
group by time
order by nPlayTimes desc;

df=pd.read_csv('q3.csv')

df.head(3)


I am working on the file "SS82_filtering/.ipynb_checkpoints/Untitled-checkpoint.ipynb".
The first task is:
Filtered catalog
And here is the final catalog, where rows (i.e, observations) with out of our interest (i.e, "obs_chunk == NaN") were removed. This catalog is written to 'Swift_Master_Stripe82_groups_filtered.csv'.
Can you write Python code for it?

cat_filtered = cat_with_obsChunksFlag.dropna(subset=['obs_chunk'])

cat_filtered

cat_filtered.describe(include='all')

cat_filtered.to_csv('Swift_Master_Stripe82_groups_filtered.csv')


I am working on the file "SS82_filtering/.ipynb_checkpoints/Untitled-checkpoint.ipynb".
The first task is:
Filtering the data
First, a closer look to an example
To have a better idea of what we should find regarding the observation time of these sources, I'll take a particular one -- V1647ORI -- and see what we have for this source.
Can you write Python code for it?

print "Table 4: Observation carried out for source 'V1647ORI' sorted in time"
g = cat_grouped_by_target.get_group('V1647ORI')
g_sorted = g.sort_values('start_time')
g_sorted


I am working on the file "SS82_filtering/.ipynb_checkpoints/Untitled-checkpoint.ipynb".
The first task is:
If we consider each group of observations of our interest -- let me call them "chunk" -- observations that distance each other no more than "X" days (for example, X=20 days) we see from this example that it happens to exist more than one "chunk" of observations per object. Here, for instance, rows 347,344,343,346 and 338,339,336,335,341 form the cluster of observations of our interest, "chunk-1" and "chunk-2", respectively.
To select the candidates we need to run a window function over the 'start_time' sorted list, where the function has two elements (i.e, observations) to ask their distance in time. If the pair of observations is less than, say 20 days, they are selected for future processing.
Applying the filter to all objects
Now defining a 20 days window as the selection criterium to all objects in our catalog we end up with 2254 observations, done over 320 objects.
Table 5 add such information through column "obs_chunk", where "Not-Available" value means the observations that have not succeed in the filtering applied.
Note: obs_chunk values mean the groupings -- "chunks" -- formed within each object's set of observations. They are unique among each object's observations, but not accross the entire catalog.
Can you write Python code for it?

def find_clustered_observations(sorted_target_observations,time_range=10):
    # Let's select a 'time_range' days window to select valid observations
    window_size = time_range

    g_sorted = sorted_target_observations

    # an ordered dictionary works as a 'set' structure
    from collections import OrderedDict
    selected_allObs = OrderedDict()

    # define en identificator for each cluster of observations, to ease future filtering
    group_obs = 1

    _last_time = None
    _last_id = None
    for _row in g_sorted.iterrows():
        ind,row = _row
        if _last_time is None:
            _last_time = row.start_time
            _last_id = ind
            continue
        _delta = row.start_time - _last_time
        if _delta.days <= window_size:
            selected_allObs[_last_id] = group_obs
            selected_allObs[ind] = group_obs
        else:
            if len(selected_allObs):
                group_obs = selected_allObs.values()[-1] + 1
        _last_time = row.start_time
        _last_id = ind

    return selected_allObs

from collections import OrderedDict
obs_indx = OrderedDict()
for name,group in cat_grouped_by_target:
    g_sorted = group.sort_values('start_time')
    filtered_indxs = find_clustered_observations(g_sorted,time_range=20)
    obs_indx.update(filtered_indxs)
import pandas
obsChunks_forFilteringCat = pandas.DataFrame(obs_indx.values(),columns=['obs_chunk'],index=obs_indx.keys())

# obsChunks_forFilteringCat.sort_index()

print "Table 5: original catalog with column 'obs_chunk' to flag which rows succeed the filtering (non-NA values)."
cat_with_obsChunksFlag = cat.join(obsChunks_forFilteringCat)
cols = list(cat_with_obsChunksFlag.columns)
cols.insert(2,cols.pop(-1))
cat_with_obsChunksFlag = cat_with_obsChunksFlag.ix[:,cols]
cat_with_obsChunksFlag


I am working on the file "SS82_filtering/.ipynb_checkpoints/Untitled-checkpoint.ipynb".
The first task is:
Target_Name is the name of the (central) object at each observation, from that we see we have 681 unique sources out of the 3035 observations. GroupSize is the number of overlapping observations, the average number is ~54. Let's see how sparse are the observations in time and how do they distribute for each source.
Can you write Python code for it?

cat['start_time'] = pandas.to_datetime(cat['start_time'])

cat_grouped_by_target = cat[['Target_Name','start_time']].groupby(['Target_Name'])

cat_descr = cat_grouped_by_target.describe().unstack()
cat_time = cat_descr.sort_values([('start_time','count')],ascending=False)
del cat_descr


I am working on the file "SS82_filtering/SS82_first_observations_filtering.ipynb".
The first task is:
If we consider each group of observations of our interest -- let me call them "chunk" -- observations that distance each other no more than "X" days (for example, X=20 days) we see from this example that it happens to exist more than one "chunk" of observations per object. Here, for instance, rows 347,344,343,346 and 338,339,336,335,341 form the cluster of observations of our interest, "chunk-1" and "chunk-2", respectively.
To select the candidates we need to run a window function over the 'start_time' sorted list, where the function has two elements (i.e, observations) to ask their distance in time. If the pair of observations is less than, say 20 days, they are selected for future processing.
Applying the filter to all objects
Now defining a 20 days window as the selection criterium to all objects in our catalog we end up with 2254 observations, done over 320 objects.
Table 5 add such information through column "obs_chunk", where "Not-Available" value means the observations that have not succeed in the filtering applied.
Note: obs_chunk values mean the groupings -- "chunks" -- formed within each object's set of observations. They are unique among each object's observations, but not accross the entire catalog.
Can you write Python code for it?

def find_clustered_observations(sorted_target_observations,time_range=10):
    # Let's select a 'time_range' days window to select valid observations
    window_size = time_range

    g_sorted = sorted_target_observations

    # an ordered dictionary works as a 'set' structure
    from collections import OrderedDict
    selected_allObs = OrderedDict()

    # define en identificator for each cluster of observations, to ease future filtering
    group_obs = 1

    _last_time = None
    _last_id = None
    for _row in g_sorted.iterrows():
        ind,row = _row
        if _last_time is None:
            _last_time = row.start_time
            _last_id = ind
            continue
        _delta = row.start_time - _last_time
        if _delta.days <= window_size:
            selected_allObs[_last_id] = group_obs
            selected_allObs[ind] = group_obs
        else:
            if len(selected_allObs):
                group_obs = selected_allObs.values()[-1] + 1
        _last_time = row.start_time
        _last_id = ind

    return selected_allObs

from collections import OrderedDict
obs_indx = OrderedDict()
for name,group in cat_grouped_by_target:
    g_sorted = group.sort_values('start_time')
    filtered_indxs = find_clustered_observations(g_sorted,time_range=20)
    obs_indx.update(filtered_indxs)
import pandas
obsChunks_forFilteringCat = pandas.DataFrame(obs_indx.values(),columns=['obs_chunk'],index=obs_indx.keys())

# obsChunks_forFilteringCat.sort_index()

print "Table 5: original catalog with column 'obs_chunk' to flag which rows succeed the filtering (non-NA values)."
cat_with_obsChunksFlag = cat.join(obsChunks_forFilteringCat)
# cols = list(cat_with_obsChunksFlag.columns)
# cols.insert(2,cols.pop(-1))
# cat_with_obsChunksFlag = cat_with_obsChunksFlag.ix[:,cols]
cat_with_obsChunksFlag


I am working on the file "SS82_filtering/SS82_first_observations_filtering.ipynb".
The first task is:
Target_Name is the name of the (central) object at each observation, from that we see we have 681 unique sources out of the 3035 observations. GroupSize is the number of overlapping observations, the average number is ~54. Let's see how sparse are the observations in time and how do they distribute for each source.
Can you write Python code for it?


I am working on the file "SS82_filtering/SS82_first_observations_filtering.ipynb".
The first task is:
The base catalog
Right below, in Table 1, we can see a sample of the catalog (the first/last five lines).
In Table 2, we see a brief description of the catalog.
Can you write Python code for it?

import pandas
cat = pandas.read_csv('Swift_Master_Stripe82_groups.ascii',
                     delim_whitespace=True)

print "Table 1: Sample of the catalog"
pandas.concat([cat.head(5),cat.tail(5)])

print "Table 2: Summary of the catalog columns"
cat.describe(include='all')


I am working on the file "SSG-bike_sharing_neural_network.ipynb".
The first task is:
Checking out the data
This dataset has the number of riders for each hour of each day from January 1 2011 to December 31 2012. The number of riders is split between casual and registered, summed up in the cnt column. You can see the first few rows of the data above.
Below is a plot showing the number of bike riders over the first 10 days in the data set. You can see the hourly rentals here. This data is pretty complicated! The weekends have lower over all ridership and there are spikes when people are biking to and from work during the week. Looking at the data above, we also have information about temperature, humidity, and windspeed, all of these likely affecting the number of riders. You'll be trying to capture all this with your model.
Dummy variables
Here we have some categorical variables like season, weather, month. To include these in our model, we'll need to make binary dummy variables. This is simple to do with Pandas thanks to get_dummies().
Can you write Python code for it?

dummy_fields = ['season', 'weathersit', 'mnth', 'hr', 'weekday']
for each in dummy_fields:
    dummies = pd.get_dummies(rides[each], prefix=each, drop_first=False)
    rides = pd.concat([rides, dummies], axis=1)

fields_to_drop = ['instant', 'dteday', 'season', 'weathersit',
                  'weekday', 'atemp', 'mnth', 'workingday', 'hr']
data = rides.drop(fields_to_drop, axis=1)
data.head()

data.info()


I am working on the file "SSG-bike_sharing_neural_network.ipynb".
The first task is:
Load and prepare the data
A critical step in working with neural networks is preparing the data correctly. Variables on different scales make it difficult for the network to efficiently learn the correct weights. Below, we've written the code to load and prepare the data. You'll learn more about this soon!
Can you write Python code for it?

data_path = 'Bike-Sharing-Dataset/hour.csv'

rides = pd.read_csv(data_path)

rides.sample()

rides.info()


I am working on the file "SSG-bike_sharing_neural_network.ipynb".
The first task is:
Scaling target variables
To make training the network easier, we'll standardize each of the continuous variables. That is, we'll shift and scale the variables such that they have zero mean and a standard deviation of 1.
The scaling factors are saved so we can go backwards when we use the network for predictions.
Can you write Python code for it?

quant_features = ['casual', 'registered', 'cnt', 'temp', 'hum', 'windspeed']
# Store scalings in a dictionary so we can convert back later
scaled_features = {}
for each in quant_features:
    mean, std = data[each].mean(), data[each].std()
    scaled_features[each] = [mean, std]
    data.loc[:, each] = (data[each] - mean)/std


I am working on the file "SSG-bike_sharing_neural_network.ipynb".
The first task is:
Splitting the data into training, testing, and validation sets
We'll save the last 21 days of the data to use as a test set after we've trained the network. We'll use this set to make predictions and compare them with the actual number of riders.
Can you write Python code for it?

# Save the last 21 days
test_data = data[-21*24:]
data = data[:-21*24]

# Separate the data into features and targets
target_fields = ['cnt', 'casual', 'registered']
features, targets = data.drop(target_fields, axis=1), data[target_fields]
test_features, test_targets = test_data.drop(target_fields, axis=1), test_data[target_fields]


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Build the Neural Network
Apply the functions you implemented above to:

Apply embedding to the input data for the encoder.
Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob).
Process target data using your process_decoding_input(target_data, target_vocab_to_int, batch_size) function.
Apply embedding to the target data for the decoder.
Decode the encoded input using your decoding_layer(dec_embed_input, dec_embeddings, encoder_state, vocab_size, sequence_length, rnn_size, num_layers, target_vocab_to_int, keep_prob).
Can you write Python code for it?

def seq2seq_model(input_data, target_data, keep_prob, batch_size, sequence_length, source_vocab_size, target_vocab_size,
                  enc_embedding_size, dec_embedding_size, rnn_size, num_layers, target_vocab_to_int):
    """
    Build the Sequence-to-Sequence part of the neural network
    :param input_data: Input placeholder
    :param target_data: Target placeholder
    :param keep_prob: Dropout keep probability placeholder
    :param batch_size: Batch Size
    :param sequence_length: Sequence Length
    :param source_vocab_size: Source vocabulary size
    :param target_vocab_size: Target vocabulary size
    :param enc_embedding_size: Decoder embedding size
    :param dec_embedding_size: Encoder embedding size
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: Tuple of (Training Logits, Inference Logits)
    """
    # TODO: Implement Function
    encoder_input = tf.contrib.layers.embed_sequence(input_data, source_vocab_size, enc_embedding_size)
    encoder_layer = encoding_layer(encoder_input, rnn_size, num_layers,keep_prob)

    decoder_input = process_decoding_input(target_data, target_vocab_to_int, batch_size)
    decoder_embed = tf.Variable(tf.random_uniform([target_vocab_size, dec_embedding_size], minval=0))

    target_embed = tf.nn.embedding_lookup(decoder_embed, decoder_input)

    tr_logits, inf_logits = decoding_layer(target_embed, decoder_embed, encoder_layer, target_vocab_size,
                                              sequence_length, rnn_size, num_layers, target_vocab_to_int,
                                              keep_prob)

    return tr_logits, inf_logits

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_seq2seq_model(seq2seq_model)


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Check Point
This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np
import helper

(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Check the Version of TensorFlow and Access to GPU
This will check to make sure you have the correct version of TensorFlow and access to a GPU
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from distutils.version import LooseVersion
import warnings
import tensorflow as tf

# Check TensorFlow Version
assert LooseVersion(tf.__version__) >= LooseVersion('1.0'), 'Please use TensorFlow version 1.0 or newer'
print('TensorFlow Version: {}'.format(tf.__version__))

# Check for a GPU
if not tf.test.gpu_device_name():
    warnings.warn('No GPU found. Please use a GPU to train your neural network.')
else:
    print('Default GPU Device: {}'.format(tf.test.gpu_device_name()))


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Checkpoint
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, (source_vocab_to_int, target_vocab_to_int), (source_int_to_vocab, target_int_to_vocab) = helper.load_preprocess()
load_path = helper.load_params()


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Decoding - Inference
Create inference logits using tf.contrib.seq2seq.simple_decoder_fn_inference() and tf.contrib.seq2seq.dynamic_rnn_decoder().
Can you write Python code for it?

def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id,
                         maximum_length, vocab_size, decoding_scope, output_fn, keep_prob):
    """
    Create a decoding layer for inference
    :param encoder_state: Encoder state
    :param dec_cell: Decoder RNN Cell
    :param dec_embeddings: Decoder embeddings
    :param start_of_sequence_id: GO ID
    :param end_of_sequence_id: EOS Id
    :param maximum_length: Maximum length of
    :param vocab_size: Size of vocabulary
    :param decoding_scope: TensorFlow Variable Scope for decoding
    :param output_fn: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: Inference Logits
    """
    # TODO: Implement Function
    decoder_fn = tf.contrib.seq2seq.simple_decoder_fn_inference(output_fn = output_fn,
                                                                encoder_state = encoder_state,
                                                                embeddings = dec_embeddings,
                                                                start_of_sequence_id = start_of_sequence_id,
                                                                end_of_sequence_id = end_of_sequence_id,
                                                                maximum_length = maximum_length,
                                                                num_decoder_symbols = vocab_size,
                                                                name = 'inference_decoder')

    outputs,_,__ = tf.contrib.seq2seq.dynamic_rnn_decoder(cell = dec_cell,
                                                          decoder_fn = decoder_fn,
                                                          scope=decoding_scope,
                                                          name='inference_decoder_rnn')
    return outputs

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_infer(decoding_layer_infer)


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Decoding - Training
Create training logits using tf.contrib.seq2seq.simple_decoder_fn_train() and tf.contrib.seq2seq.dynamic_rnn_decoder().  Apply the output_fn to the tf.contrib.seq2seq.dynamic_rnn_decoder() outputs.
Can you write Python code for it?

def decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length, decoding_scope,
                         output_fn, keep_prob):
    """
    Create a decoding layer for training
    :param encoder_state: Encoder State
    :param dec_cell: Decoder RNN Cell
    :param dec_embed_input: Decoder embedded input
    :param sequence_length: Sequence Length
    :param decoding_scope: TenorFlow Variable Scope for decoding
    :param output_fn: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: Train Logits
    """
    # TODO: Implement Function
    decoder = tf.contrib.seq2seq.simple_decoder_fn_train(encoder_state, name='decoder_train')
    decoder_drop = tf.contrib.rnn.DropoutWrapper(dec_cell, keep_prob)
    decoder_outputs,_,__ = tf.contrib.seq2seq.dynamic_rnn_decoder(decoder_drop, decoder,
                                                                  inputs = dec_embed_input,
                                                                  sequence_length = sequence_length,
                                                                  scope = decoding_scope)

    return output_fn(decoder_outputs)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_train(decoding_layer_train)


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Encoding
Implement encoding_layer() to create a Encoder RNN layer using tf.nn.dynamic_rnn().
Can you write Python code for it?

def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob):
    """
    Create encoding layer
    :param rnn_inputs: Inputs for the RNN
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param keep_prob: Dropout keep probability
    :return: RNN state
    """
#     TODO: Implement Function
    lstm_cell = tf.contrib.rnn.BasicLSTMCell(num_units = rnn_size)

    cell = tf.contrib.rnn.MultiRNNCell([lstm_cell] * num_layers)

    state_0, state_1 = tf.nn.dynamic_rnn(cell, rnn_inputs, dtype=tf.float32)

    return state_1

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_encoding_layer(encoding_layer)


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Explore the Data
Play around with view_sentence_range to view different parts of the data.
Can you write Python code for it?

view_sentence_range = (1, 10)

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import numpy as np

print('Dataset Stats')
print('Roughly the number of unique words: {}'.format(len({word: None for word in source_text.split()})))

sentences = source_text.split('\n')
word_counts = [len(sentence.split()) for sentence in sentences]
print('Number of sentences: {}'.format(len(sentences)))
print('Average number of words in a sentence: {}'.format(np.average(word_counts)))

print()
print('English sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(source_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))
print()
print('French sentences {} to {}:'.format(*view_sentence_range))
print('\n'.join(target_text.split('\n')[view_sentence_range[0]:view_sentence_range[1]]))


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Implement Preprocessing Function
Text to Word Ids
As you did with other RNNs, you must turn the text into a number so the computer can understand it. In the function text_to_ids(), you'll turn source_text and target_text from words to ids.  However, you need to add the <EOS> word id at the end of each sentence from target_text.  This will help the neural network predict when the sentence should end.
You can get the <EOS> word id by doing:
python
target_vocab_to_int['<EOS>']
You can get other word ids using source_vocab_to_int and target_vocab_to_int.
Can you write Python code for it?

def text_to_ids(source_text, target_text, source_vocab_to_int, target_vocab_to_int):
    """
    Convert source and target text to proper word ids
    :param source_text: String that contains all the source text.
    :param target_text: String that contains all the target text.
    :param source_vocab_to_int: Dictionary to go from the source words to an id
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: A tuple of lists (source_id_text, target_id_text)
    """
    s_sentences = source_text.split('\n')
    t_sentences = [sentence + ' <EOS>' for sentence in target_text.split('\n')]
    source_ids = [[source_vocab_to_int[word] for word in line.split()] for line in s_sentences]
    target_ids = [[target_vocab_to_int[word] for word in line.split()] for line in t_sentences]

    return (source_ids, target_ids)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_text_to_ids(text_to_ids)


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Preprocess all the data and save it
Running the code cell below will preprocess all the data and save it to file.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
helper.preprocess_and_save_data(source_path, target_path, text_to_ids)


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Process Decoding Input
Implement process_decoding_input using TensorFlow to remove the last word id from each batch in target_data and concat the GO ID to the begining of each batch.
Can you write Python code for it?

def process_decoding_input(target_data, target_vocab_to_int, batch_size):
    """
    Preprocess target data for dencoding
    :param target_data: Target Placehoder
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param batch_size: Batch Size
    :return: Preprocessed target data
    """
    # TODO: Implement Function

    go = target_vocab_to_int['<GO>']
    target_data = tf.strided_slice(target_data, [0,0], [batch_size, -1], [1,1])
    aux_data = tf.fill([batch_size, 1], go)
    target_data = tf.concat([aux_data, target_data],1)
    return target_data

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_process_decoding_input(process_decoding_input)


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Save Parameters
Save the batch_size and save_path parameters for inference.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Save parameters for checkpoint
helper.save_params(save_path)


I am working on the file "SSG_dlnd_language_translation_.ipynb".
The first task is:
Train
Train the neural network on the preprocessed data. If you have a hard time getting a good loss, check the forms to see if anyone is having the same problem.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import time

def get_accuracy(target, logits):
    """
    Calculate accuracy
    """
    max_seq = max(target.shape[1], logits.shape[1])
    if max_seq - target.shape[1]:
        target = np.pad(
            target_batch,
            [(0,0),(0,max_seq - target_batch.shape[1]), (0,0)],
            'constant')
    if max_seq - batch_train_logits.shape[1]:
        logits = np.pad(
            logits,
            [(0,0),(0,max_seq - logits.shape[1]), (0,0)],
            'constant')

    return np.mean(np.equal(target, np.argmax(logits, 2)))

train_source = source_int_text[batch_size:]
train_target = target_int_text[batch_size:]

valid_source = helper.pad_sentence_batch(source_int_text[:batch_size])
valid_target = helper.pad_sentence_batch(target_int_text[:batch_size])

with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(epochs):
        for batch_i, (source_batch, target_batch) in enumerate(
                helper.batch_data(train_source, train_target, batch_size)):
            start_time = time.time()

            _, loss = sess.run(
                [train_op, cost],
                {input_data: source_batch,
                 targets: target_batch,
                 lr: learning_rate,
                 sequence_length: target_batch.shape[1],
                 keep_prob: keep_probability})

            batch_train_logits = sess.run(
                inference_logits,
                {input_data: source_batch, keep_prob: 1.0})
            batch_valid_logits = sess.run(
                inference_logits,
                {input_data: valid_source, keep_prob: 1.0})

            train_acc = get_accuracy(target_batch, batch_train_logits)
            valid_acc = get_accuracy(np.array(valid_target), batch_valid_logits)
            end_time = time.time()
            print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.3f}, Validation Accuracy: {:>6.3f}, Loss: {:>6.3f}'
      ...


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Batches
Implement get_batches to create batches of input and targets using int_text.  The batches should be a Numpy array with the shape (number of batches, 2, batch size, sequence length). Each batch contains two elements:
- The first element is a single batch of input with the shape [batch size, sequence length]
- The second element is a single batch of targets with the shape [batch size, sequence length]
If you can't fill the last batch with enough data, drop the last batch.
For exmple, get_batches([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], 2, 3) would return a Numpy array of the following:
```
[
  # First Batch
  [
    # Batch of Input
    [[ 1  2  3], [ 7  8  9]],
    # Batch of targets
    [[ 2  3  4], [ 8  9 10]]
  ],
# Second Batch
  [
    # Batch of Input
    [[ 4  5  6], [10 11 12]],
    # Batch of targets
    [[ 5  6  7], [11 12 13]]
  ]
]
```
Can you write Python code for it?

def get_batches(int_text, batch_size, seq_length):

    n_batches = int(len(int_text) / (batch_size * seq_length))

    # Drop the last few characters to make only full batches
    data_x = np.array(int_text[: n_batches * batch_size * seq_length])
    data_y = np.array(int_text[1: n_batches * batch_size * seq_length + 1])

    batches_x = np.split(data_x.reshape(batch_size, -1), n_batches, 1)
    batches_y = np.split(data_y.reshape(batch_size, -1), n_batches, 1)

    return np.array(list(zip(batches_x, batches_y)))
"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_batches(get_batches)


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Build RNN
You created a RNN Cell in the get_init_cell() function.  Time to use the cell to create a RNN.
- Build the RNN using the tf.nn.dynamic_rnn()
 - Apply the name "final_state" to the final state using tf.identity()
Return the outputs and final_state state in the following tuple (Outputs, FinalState)
Can you write Python code for it?

def build_rnn(cell, inputs):
    """
    Create a RNN using a RNN Cell
    :param cell: RNN Cell
    :param inputs: Input text data
    :return: Tuple (Outputs, Final State)
    """
    outputs, state = tf.nn.dynamic_rnn(cell, inputs, dtype = tf.float32)
    final_state = tf.identity(state, name = 'final_state')
    return outputs, final_state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_build_rnn(build_rnn)


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Build the Graph
Build the graph using the neural network you implemented.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
from tensorflow.contrib import seq2seq

train_graph = tf.Graph()
with train_graph.as_default():
    vocab_size = len(int_to_vocab)
    input_text, targets, lr = get_inputs()
    input_data_shape = tf.shape(input_text)
    cell, initial_state = get_init_cell(input_data_shape[0], rnn_size)
    logits, final_state = build_nn(cell, rnn_size, input_text, vocab_size)

    # Probabilities for generating words
    probs = tf.nn.softmax(logits, name='probs')

    # Loss function
    cost = seq2seq.sequence_loss(
        logits,
        targets,
        tf.ones([input_data_shape[0], input_data_shape[1]]))

    # Optimizer
    optimizer = tf.train.AdamOptimizer(lr)

    # Gradient Clipping
    gradients = optimizer.compute_gradients(cost)
    capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients]
    train_op = optimizer.apply_gradients(capped_gradients)


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Build the Neural Network
Apply the functions you implemented above to:
- Apply embedding to input_data using your get_embed(input_data, vocab_size, embed_dim) function.
- Build RNN using cell and your build_rnn(cell, inputs) function.
- Apply a fully connected layer with a linear activation and vocab_size as the number of outputs.
Return the logits and final state in the following tuple (Logits, FinalState)
Can you write Python code for it?

def build_nn(cell, rnn_size, input_data, vocab_size):
    """
    Build part of the neural network
    :param cell: RNN cell
    :param rnn_size: Size of rnns
    :param input_data: Input data
    :param vocab_size: Vocabulary size
    :return: Tuple (Logits, FinalState)
    """
    # TODO: Implement Function
    embedding = get_embed(input_data, vocab_size, rnn_size)
    outputs, final_state = build_rnn(cell, embedding)
    logits = tf.contrib.layers.fully_connected(outputs, vocab_size, activation_fn=None)

    return (logits, final_state)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_build_nn(build_nn)


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Build the Neural Network
You'll build the components necessary to build a RNN by implementing the following functions below:
- get_inputs
- get_init_cell
- get_embed
- build_rnn
- build_nn
- get_batches
Check the Version of TensorFlow and Access to GPU
Can you write Python code for it?


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Check Point
This is your first checkpoint. If you ever decide to come back to this notebook or have to restart the notebook, you can start from here. The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import helper
import numpy as np
import problem_unittests as tests

int_text, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Checkpoint
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import tensorflow as tf
import numpy as np
import helper
import problem_unittests as tests

_, vocab_to_int, int_to_vocab, token_dict = helper.load_preprocess()
seq_length, load_dir = helper.load_params()


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Choose Word
Implement the pick_word() function to select the next word using probabilities.
Can you write Python code for it?

def pick_word(probabilities, int_to_vocab):
    """
    Pick the next word in the generated text
    :param probabilities: Probabilites of the next word
    :param int_to_vocab: Dictionary of word ids as the keys and words as the values
    :return: String of the predicted word
    """
#     i = np.random.choice(np.arange(len(int_to_vocab)), probabilities)

    probs = list(probabilities)
#     print(len(probabilities), len(int_to_vocab))
    i = np.random.choice(np.arange(len(int_to_vocab)), p=probabilities)
    return int_to_vocab[i]

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_pick_word(pick_word)


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Generate TV Script
This will generate the TV script for you.  Set gen_length to the length of TV script you want to generate.
Can you write Python code for it?

gen_length = 200
# homer_simpson, moe_szyslak, or Barney_Gumble
prime_word = 'moe_szyslak'

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_dir + '.meta')
    loader.restore(sess, load_dir)

    # Get Tensors from loaded model
    input_text, initial_state, final_state, probs = get_tensors(loaded_graph)

    # Sentences generation setup
    gen_sentences = [prime_word + ':']
    prev_state = sess.run(initial_state, {input_text: np.array([[1]])})

    # Generate sentences
    for n in range(gen_length):
        # Dynamic Input
        dyn_input = [[vocab_to_int[word] for word in gen_sentences[-seq_length:]]]
        dyn_seq_length = len(dyn_input[0])

        # Get Prediction
        probabilities, prev_state = sess.run(
            [probs, final_state],
            {input_text: dyn_input, initial_state: prev_state})

        pred_word = pick_word(probabilities[dyn_seq_length-1], int_to_vocab)

        gen_sentences.append(pred_word)

    # Remove tokens
    tv_script = ' '.join(gen_sentences)
    for key, token in token_dict.items():
        ending = ' ' if key in ['\n', '(', '"'] else ''
        tv_script = tv_script.replace(' ' + token.lower(), key)
    tv_script = tv_script.replace('\n ', '\n')
    tv_script = tv_script.replace('( ', '(')

    print(tv_script)


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Implement Generate Functions
Get Tensors
Get tensors from loaded_graph using the function get_tensor_by_name().  Get the tensors using the following names:
- "input:0"
- "initial_state:0"
- "final_state:0"
- "probs:0"
Return the tensors in the following tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)
Can you write Python code for it?

def get_tensors(loaded_graph):
    """
    Get input, initial state, final state, and probabilities tensor from <loaded_graph>
    :param loaded_graph: TensorFlow graph loaded from file
    :return: Tuple (InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor)
    """
    # TODO: Implement Function
    InputTensor = loaded_graph.get_tensor_by_name('input:0')
    InitialStateTensor = loaded_graph.get_tensor_by_name('initial_state:0')
    FinalStateTensor = loaded_graph.get_tensor_by_name('final_state:0')
    ProbsTensor = loaded_graph.get_tensor_by_name('probs:0')

    return InputTensor, InitialStateTensor, FinalStateTensor, ProbsTensor

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_tensors(get_tensors)


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Implement Preprocessing Functions
The first thing to do to any dataset is preprocessing.  Implement the following preprocessing functions below:
- Lookup Table
- Tokenize Punctuation
Lookup Table
To create a word embedding, you first need to transform the words to ids.  In this function, create two dictionaries:
- Dictionary to go from the words to an id, we'll call vocab_to_int
- Dictionary to go from the id to word, we'll call int_to_vocab
Return these dictionaries in the following tuple (vocab_to_int, int_to_vocab)
Can you write Python code for it?

import numpy as np
import problem_unittests as tests
from collections import Counter
def create_lookup_tables(text):

    counts = Counter(text)
    vocab = sorted(counts, key=counts.get, reverse=True)
    vocab_to_int = {word: w for w, word in enumerate(vocab)}
    int_to_vocab = {i:j for j, i in vocab_to_int.items()}
    return (vocab_to_int, int_to_vocab)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_create_lookup_tables(create_lookup_tables)


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Neural Network Training
Hyperparameters
Tune the following parameters:

Set num_epochs to the number of epochs.
Set batch_size to the batch size.
Set rnn_size to the size of the RNNs.
Set seq_length to the length of sequence.
Set learning_rate to the learning rate.
Set show_every_n_batches to the number of batches the neural network should print progress.
Can you write Python code for it?

# Number of Epochs
num_epochs = 65
# Batch Size
batch_size = 64
# RNN Size
rnn_size = 256
# Sequence Length
seq_length = 15
# Learning Rate
learning_rate = 0.005
# Show stats for every n number of batches
show_every_n_batches = 71

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
save_dir = './save'


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Preprocess all the data and save it
Running the code cell below will preprocess all the data and save it to file.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Preprocess Training, Validation, and Testing Data
helper.preprocess_and_save_data(data_dir, token_lookup, create_lookup_tables)


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Save Parameters
Save seq_length and save_dir for generating a new TV script.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
# Save parameters for checkpoint
helper.save_params((seq_length, save_dir))


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Tokenize Punctuation
We'll be splitting the script into a word array using spaces as delimiters.  However, punctuations like periods and exclamation marks make it hard for the neural network to distinguish between the word "bye" and "bye!".
Implement the function token_lookup to return a dict that will be used to tokenize symbols like "!" into "||Exclamation_Mark||".  Create a dictionary for the following symbols where the symbol is the key and value is the token:
- Period ( . )
- Comma ( , )
- Quotation Mark ( " )
- Semicolon ( ; )
- Exclamation mark ( ! )
- Question mark ( ? )
- Left Parentheses ( ( )
- Right Parentheses ( ) )
- Dash ( -- )
- Return ( \n )
This dictionary will be used to token the symbols and add the delimiter (space) around it.  This separates the symbols as it's own word, making it easier for the neural network to predict on the next word. Make sure you don't use a token that could be confused as a word. Instead of using the token "dash", try using something like "||dash||".
Can you write Python code for it?

def token_lookup():

    token_dict = {'.':'||period||',
                  ',':'||comma||',
                  '"':'||quotm||',
                  ';':'||semic||',
                  '!':'||exclMark||',
                  '?':'||qMark||',
                  '(':'||lpar||',
                  ')':'||rpar||',
                  '--':'||dash||',
                  '\n':'||ret||'}
    return token_dict

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_tokenize(token_lookup)


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Train
Train the neural network on the preprocessed data.  If you have a hard time getting a good loss, check the forms to see if anyone is having the same problem.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
batches = get_batches(int_text, batch_size, seq_length)

with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(num_epochs):
        state = sess.run(initial_state, {input_text: batches[0][0]})

        for batch_i, (x, y) in enumerate(batches):
            feed = {
                input_text: x,
                targets: y,
                initial_state: state,
                lr: learning_rate}
            train_loss, state, _ = sess.run([cost, final_state, train_op], feed)

            # Show every <show_every_n_batches> batches
            if (epoch_i * len(batches) + batch_i) % show_every_n_batches == 0:
                print('Epoch {:>3} Batch {:>4}/{}   train_loss = {:.3f}'.format(
                    epoch_i,
                    batch_i,
                    len(batches),
                    train_loss))

    # Save Model
    saver = tf.train.Saver()
    saver.save(sess, save_dir)
    print('Model Trained and Saved')


I am working on the file "SSG_dlnd_tv_script_generation.ipynb".
The first task is:
Word Embedding
Apply embedding to input_data using TensorFlow.  Return the embedded sequence.
Can you write Python code for it?

def get_embed(input_data, vocab_size, embed_dim):

    embedding = tf.Variable(tf.random_uniform((vocab_size, embed_dim), -1, 1))
    embed = tf.nn.embedding_lookup(embedding, input_data)
    return embed

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_get_embed(get_embed)


I am working on the file "SSG_image_classification_DNN.ipynb".
The first task is:
Build the network
For the neural network, you'll build each layer into a function.  Most of the code you've seen has been outside of functions. To test your code more thoroughly, we require that you put each layer in a function.  This allows us to give you better feedback and test for simple mistakes using our unittests before you submit your project.

Note: If you're finding it hard to dedicate enough time for this course each week, we've provided a small shortcut to this part of the project. In the next couple of problems, you'll have the option to use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages to build each layer, except the layers you build in the "Convolutional and Max Pooling Layer" section.  TF Layers is similar to Keras's and TFLearn's abstraction to layers, so it's easy to pickup.
However, if you would like to get the most out of this course, try to solve all the problems without using anything from the TF Layers packages. You can still use classes from other packages that happen to have the same name as ones you find in TF Layers! For example, instead of using the TF Layers version of the conv2d class, tf.layers.conv2d, you would want to use the TF Neural Network version of conv2d, tf.nn.conv2d.

Let's begin!
Input
The neural network needs to read the image data, one-hot encoded labels, and dropout keep probability. Implement the following functions
* Implement neural_net_image_input
 * Return a TF Placeholder
 * Set the shape using image_shape with batch size set to None.
 * Name the TensorFlow placeholder "x" using the TensorFlow name parameter in the TF Placeholder.
* Implement neural_net_label_input
 * Return a TF Placeholder
 * Set the shape using n_classes with batch size set to None.
 * Name the TensorFlow placeholder "y" using the TensorFlow name parameter in the TF Placeholder.
* Implement neural_net_keep_prob_input
 * Return a TF Placeholder for dropout keep probability.
 * Name the TensorFlow placeholder "keep_prob" using the TensorFlow name parameter in...
Can you write Python code for it?

import tensorflow as tf

def neural_net_image_input(image_shape):
    """
    Return a Tensor for a bach of image input
    : image_shape: Shape of the images
    : return: Tensor for image input.
    """
    return tf.placeholder(tf.float32, shape=[None, image_shape[0], image_shape[1], image_shape[2]], name='x')

def neural_net_label_input(n_classes):
    """
    Return a Tensor for a batch of label input
    : n_classes: Number of classes
    : return: Tensor for label input.
    """
    return tf.placeholder(tf.float32, shape=[None, n_classes], name='y')

def neural_net_keep_prob_input():
    """
    Return a Tensor for keep probability
    : return: Tensor for keep probability.
    """
    return tf.placeholder(tf.float32, name='keep_prob')

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tf.reset_default_graph()
tests.test_nn_image_inputs(neural_net_image_input)
tests.test_nn_label_inputs(neural_net_label_input)
tests.test_nn_keep_prob_inputs(neural_net_keep_prob_input)


I am working on the file "SSG_image_classification_DNN.ipynb".
The first task is:
Check Point
This is your first checkpoint.  If you ever decide to come back to this notebook or have to restart the notebook, you can start from here.  The preprocessed data has been saved to disk.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import pickle
import problem_unittests as tests
import helper

# Load the Preprocessed Validation data
valid_features, valid_labels = pickle.load(open('preprocess_validation.p', mode='rb'))


I am working on the file "SSG_image_classification_DNN.ipynb".
The first task is:
Flatten Layer
Implement the flatten function to change the dimension of x_tensor from a 4-D tensor to a 2-D tensor.  The output should be the shape (Batch Size, Flattened Image Size). Shortcut option: you can use classes from the TensorFlow Layers or TensorFlow Layers (contrib) packages for this layer. For more of a challenge, only use other TensorFlow packages.
Can you write Python code for it?

def flatten(x_tensor):
    """
    Flatten x_tensor to (Batch Size, Flattened Image Size)
    : x_tensor: A tensor of size (Batch Size, ...), where ... are the image dimensions.
    : return: A tensor of size (Batch Size, Flattened Image Size).
    """

    return tf.contrib.layers.flatten(x_tensor)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_flatten(flatten)


I am working on the file "SSG_image_classification_DNN.ipynb".
The first task is:
Implement Preprocess Functions
Normalize
In the cell below, implement the normalize function to take in image data, x, and return it as a normalized Numpy array. The values should be in the range of 0 to 1, inclusive.  The return object should be the same shape as x.
Can you write Python code for it?

import numpy as np
def normalize(x):
    """
    Normalize a list of sample image data in the range of 0 to 1
    : x: List of image data.  The image shape is (32, 32, 3)
    : return: Numpy array of normalize data
    """
    # TODO: Implement Function
#     n = np.linalg.norm(x)
#     return x/n

    x = np.array(x)
    x_min, x_max = x.min(), x.max()

    return (x - x_min) / (x_max - x_min)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_normalize(normalize)


I am working on the file "SSG_image_classification_DNN.ipynb".
The first task is:
One-hot encode
Just like the previous code cell, you'll be implementing a function for preprocessing.  This time, you'll implement the one_hot_encode function. The input, x, are a list of labels.  Implement the function to return the list of labels as One-Hot encoded Numpy array.  The possible values for labels are 0 to 9. The one-hot encoding function should return the same encoding for each value between each call to one_hot_encode.  Make sure to save the map of encodings outside the function.
Hint: Don't reinvent the wheel.
Can you write Python code for it?

from sklearn.preprocessing import LabelBinarizer
one_hot_encoder = LabelBinarizer()
one_hot_encoder.fit(np.arange(0,10))
def one_hot_encode(x):
    """
    One hot encode a list of sample labels. Return a one-hot encoded vector for each label.
    : x: List of sample Labels
    : return: Numpy array of one-hot encoded labels
    """
    # TODO: Implement Function
    return one_hot_encoder.transform(x)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_one_hot_encode(one_hot_encode)


I am working on the file "SSG_image_classification_DNN.ipynb".
The first task is:
Show Stats
Implement the function print_stats to print loss and validation accuracy.  Use the global variables valid_features and valid_labels to calculate validation accuracy.  Use a keep probability of 1.0 to calculate the loss and validation accuracy.
Can you write Python code for it?

def print_stats(session, feature_batch, label_batch, cost, accuracy):
    """
    Print information about loss and validation accuracy
    : session: Current TensorFlow session
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    : cost: TensorFlow cost function
    : accuracy: TensorFlow accuracy function
    """
    loss = session.run(cost, feed_dict={x: feature_batch, y: label_batch, keep_prob: 1.0})
    accuracy = session.run(accuracy, feed_dict={x: valid_features, y: valid_labels, keep_prob: 1.0})

    print("Loss: {0}, Val_Acc: {1}".format(loss, accuracy))
    pass


I am working on the file "SSG_image_classification_DNN.ipynb".
The first task is:
Train on a Single CIFAR-10 Batch
Instead of training the neural network on all the CIFAR-10 batches of data, let's use a single batch. This should save time while you iterate on the model to get a better accuracy.  Once the final validation accuracy is 50% or greater, run the model on all the data in the next section.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
print('Checking the Training on a Single Batch...')
with tf.Session() as sess:
    # Initializing the variables
    sess.run(tf.global_variables_initializer())

    # Training cycle
    for epoch in range(epochs):
        batch_i = 1
        for batch_features, batch_labels in helper.load_preprocess_training_batch(batch_i, batch_size):
            train_neural_network(sess, optimizer, keep_probability, batch_features, batch_labels)
        print('Epoch {:>2}, CIFAR-10 Batch {}:  '.format(epoch + 1, batch_i), end='')
        print_stats(sess, batch_features, batch_labels, cost, accuracy)


I am working on the file "SSG_image_classification_DNN.ipynb".
The first task is:
Train the Neural Network
Single Optimization
Implement the function train_neural_network to do a single optimization.  The optimization should use optimizer to optimize in session with a feed_dict of the following:
* x for image input
* y for labels
* keep_prob for keep probability for dropout
This function will be called for each batch, so tf.global_variables_initializer() has already been called.
Note: Nothing needs to be returned. This function is only optimizing the neural network.
Can you write Python code for it?

def train_neural_network(session, optimizer, keep_probability, feature_batch, label_batch):
    """
    Optimize the session on a batch of images and labels
    : session: Current TensorFlow session
    : optimizer: TensorFlow optimizer function
    : keep_probability: keep probability
    : feature_batch: Batch of Numpy image data
    : label_batch: Batch of Numpy label data
    """
    session.run(optimizer, feed_dict={x: feature_batch, y: label_batch, keep_prob: keep_probability})
    pass

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_train_nn(train_neural_network)


I am working on the file "SSH.ipynb".
The first task is:
Bulk system in momentum space and $d$-space
First let us define the $d(k)$ vector as functions of the intracell $v$ and intercell $w$ hopping parameters.
$$\mathbf{d}(k)=\left(\begin{array}{c}
v+w\cos(k)\
w\sin(k)\
0\
\end{array}\right) $$
Can you write Python code for it?

def dk(k,v,w,**kwargs):
    '''
    This function returns the d vector of the SSH model.
    '''
    return [v+w*cos(k),w*sin(k),0]


I am working on the file "SSH.ipynb".
The first task is:
Finite system in real space, spectrum and wavefunctions
First let us define a simple function that generates the Hamiltonian of a finite tight-binding chain for given $v$ and $w$. For example if we have a system with three unitcells the Hamiltonian will look like
\begin{equation}
H_{\mathrm{SSH}}=\left(\begin{array}{cccccc}
0 & v\
v & 0 & w &  & 0\
 & w & 0 & v\
 &  & v & 0 & w\
 & 0 &  & w & 0 & v\
 &  &  &  & v & 0
\end{array}\right).
\end{equation}
Can you write Python code for it?

def H_SSH_reals(L,v,w):
    '''
    A function to bulid a finite SSH chain.
    The number of unitcells is L.
    As usual v is intracell and w ins intercell hopping.
    '''
    idL=eye(L); # identity matrix of dimension L
    odL=diag(ones(L-1),1);# upper off diagonal matrix with ones of size L
    U=matrix([[0,1],[1,0]]) # intracell
    T=matrix([[0,0],[1,0]]) # intercell
    return kron(idL,v*U)+kron(odL,w*T)+kron(odL,w*T).H


I am working on the file "SSH.ipynb".
The first task is:
Now we can write a simple interactive plot to explore the $\mathbf{d}(k)$ curve and the spectrum $E_\pm(k)=\pm\left|\mathbf{d}(k)\right|$ as we tune the hopping parameters.
Can you write Python code for it?

figsize(12,6)
kran=linspace(-pi,pi,200)
@interact(v=(-1,1,0.1),w=(-1,1,0.1))
def ekdk(v=0.5,w=0.5):
    dx,dy=dk(kran,v,w)[:2]
    #
    #-- This part makes the k-space figure--
    #
    subplot(121)
    plot(kran,sqrt(dx**2+dy**2),'k-',linewidth=3)           # This creates the
    plot(kran,-sqrt(dx**2+dy**2),'k-',linewidth=3)          # two bandlines
    #just to make it look like in the book
    ylabel('energy E',fontsize=20);
    xlabel(r'wavenumber $k$',fontsize=20);
    xlim(-pi,pi);xticks([-pi,0,pi],['$-\pi$','0','$\pi$'],fontsize=20)
    ylim(-2.02,2.02);yticks([-2,-1,0,1,2],['-2','-1','0','1','2'],fontsize=20);
    plot(kran,0*kran,'k--')
    plot([0,0],[-2.5,2.5],'k--')

    #
    #--This part makes the d-space figure--
    #
    subplot(122)
    plot(dx,dy,'k-',linewidth=3)                                # The d(k) line itself
    arrow(dx[30],dy[30],(dx[31]-dx[29])/30,(dy[31]-dy[29])/30,  # and an arrow
          head_width=0.15, head_length=0.2, fc='k', ec='k')     # showing winding direction

    #just to make it look like in the book
    plot([0],[0],'wo',markersize=20)
    if abs(v)==abs(w):                       # Here we have a simple criterion for
        plot([0],[0],'ro',markersize=10)     # metallicity
    plot([1,1],[-0.1,0.1],'k')
    plot([-1,-1],[-0.1,0.1],'k')
    plot([-0.1,0.1],[1,1],'k')
    plot([-0.1,0.1],[-1,-1],'k')
    arrow(-2.0,0,3.6,0,head_width=0.2, head_length=0.4, fc='k', ec='k')
    arrow(0,-1.5,0,2*1.4,head_width=0.2, head_length=0.4, fc='k', ec='k')
    xlim(-2.1,2.1)
    ylim(-2.1,2.1)
    axis('off')
    text(.9,-0.5,'1',fontsize=20);text(-1.3,-0.5,'-1',fontsize=20);
    text(-0.5,.9,'1',fontsize=20);text(-0.7,-1.1,'-1',fontsize=20);
    text(0.25,1.4,r'$d_y$',fontsize=20);text(2,-0.5,r'$d_x$',fontsize=20);


I am working on the file "SSH.ipynb".
The first task is:
Now we will use the above function to evaluate the spectrum and wavefunctions for $w=1$ as the function of $v$.
We shall consider a small system with only $10$ unitcells. As we will see this is more than enought to visualize all the interesting effects and yet it is small enough for quick calculations.
First we generate the eigenspectra and the wavefunctions!
Can you write Python code for it?

L=10;      # Number of unitcells to take
dat=[];
vecdat=[];
vran=linspace(0,3,100) # This array contains the v values
                       # we evaluate the spectrum for.
for v in vran:
    w=1.0;
    H=H_SSH_reals(L,v,w)
    eigdat=eigh(H);      # for a given v here vi calculate the eigensystem (values and vectors)
    dat=append(dat,eigdat[0]);
    vecdat=append(vecdat,eigdat[1]);

dat=reshape(dat,[len(vran),2*L]);          # rewraping the data
vecdat=reshape(vecdat,[len(vran),2*L,2*L]) # to be more digestable


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
A last fun example of using widgets can be found in the accompanying notebook on the Quantum double-slit experiment.
4.5 When things go awry
Making mistakes when programming is an unavoidable part of the experience.  It can be very frustrating.  You might spend an hour trying to track down the source of a "bug" in a program, convinced that the computer is at fault.  I know I have done that countless times.  But computers are only as "smart" as their programmers.  Like lemmings, they will blindly follow our instructions.  It is pretty much always our fault.  Get used to that.
Occasionally, the computer will spit out an error message or give back a gibberish answer, alerting us to our mistake.  Sometimes we're not so lucky.
Bug-hunting (or "debugging") is part of the job.  And part of debugging is being able to understand error messages.  Let's generate some errors to get a feel for dealing with them.
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Change the value of the variable with an arithmetic expression:
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Create a variable and assign it a value:
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
For repetitive tasks, you can use a for loop:
Can you write Python code for it?

print "I can count to 10!"
for i in range(10):
    print i


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Graphs in matplotlib have many attributes that you can customize (see the documentation).  The developers also provide a gallery of plots to showcase the wide variety of visual representations that are available.
No one can keep all of the functions and fine layout control commands in their brain.  Often when I need to make a plot, I go to the gallery page and browse the images until I find one that is similar to what I want to create and then I copy the code and modify it to suit my needs.
When you use the Matplotlib gallery to develop (or "template") a figure, you can very easily load the source code into your notebook and then modify it as needed to fit your specific needs using the Python "magic" command, %load.
Try it now.  After the code is loaded, just execute the cell to see the output.
Can you write Python code for it?

#Execute this cell to see the histogram example plot code, then execute again to make the plot.
%load http://matplotlib.org/mpl_examples/statistics/histogram_demo_multihist.py


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Here's another interactive plot that allows you to randomly sample (x,y) pairs within a circle of radius $r$.  The interact object lets you increase or decrease the number of samples in the circle.
Can you write Python code for it?

def scatter_plot(r=0.5, n=27):
    t = np.random.uniform(0.0,2.0*np.pi,n)
    rad = r*np.sqrt(np.random.uniform(0.0,1.0,n))
    x = np.empty(n)
    y = np.empty(n)
    x = rad*np.cos(t)
    y = rad*np.sin(t)
    fig = plt.figure(figsize=(4,4),dpi=80)
    plt.scatter(x,y)
    plt.xlim(-1.,1.)
    plt.ylim(-1.,1.)
    plt.show()

v2 = interact(scatter_plot,r=(0.0,1.0), n=(1,1000))


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
How about printing them backward?
Can you write Python code for it?

#Iterate from the last to the first element in steps of 1:
print shopping_list[-1::-1]


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
How do we avoid logical errors in general?
Never make a logical error.
Just kidding.  The only way to minimize logical errors is to always test your code with a known result whenever possible.  In the case of the cube_root function, we knew something was wrong with the logic because the cube-root of 8 should be 2.  When we got a different value than expected, we knew we had a problem to track down and correct.   The purpose of testing as a programming "best practice" is to minimize logical errors like this one.
Fix the cube_root function and test that it gives sensible results.
4.6 Getting help
When IPython needs to display additional information it will automatically invoke a pager at the bottom of the screen.  You can get help with IPython in general by typing a question mark in a code cell and executing it:
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
How do we avoid logical errors like this one?  For mathematical calculations, you can explicitly define the data type of the numbers you use.  If you want a floating-point result, use floating-point numbers by adding a decimal point to the end of the number:
Can you write Python code for it?

a = 3
b = 4
print a/b

c = 3.
d = 4.
print c/d


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
I can access individual items from their location in the list using an index to that location:
Can you write Python code for it?

print shopping_list[2]

print "What's for breakfast?"
print shopping_list[2] + shopping_list[0]


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
I can also slice through the list
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
In Python, you can define a function and pass it arguments, then execute the function:
Can you write Python code for it?

def intro(name):
    print "Hi, my name is",name


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
It is also possible to display images linked from sites around the web:
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Let's move on to another error.
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Now what?  Here is a type error.  We tried to perform the "+" operation on a string and an integer, which is not allowed.  To correct this we could force the integer object to be treated as a string by casting it as one:
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
NumPy arrays can be used for simple or complex mathematical calculations.
Can you write Python code for it?

x = np.linspace(0, 2*np.pi, 300)
y = np.sin(x**2)


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Oops.  Not quite what you were expecting?
Notice that in Python, indices start from 0, not 1, and count up to n-1.
I can create a list of items and iterate over them with a for loop:
Can you write Python code for it?

shopping_list = ["eggs","milk","bacon","bread","strawberries","yogurt","jam"]
for item in shopping_list:
    print item


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Oops.  We misspelled print, giving us another syntax error.  In this case the message is not quite as clear about what the problem is, but it isn't too hard to spot the mistake.
Notice that Python doesn't execute the second statement in the cell when it encounters an error in the first line. Execution stops at the first sign of an error.  Keeping the number of lines of code in cells or functions short makes tracking errors much easier.
Notice also that the Notebook provides syntax highlighting - Python keywords, like print, and numbers appear in green, quoted strings appear in red, operators like + and - appear in purple and variables are black.  This can be helpful for spotting errors quickly.
Let's correct our spelling.
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Or get help on a particular function with, e.g.
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Print the value stored in the variable:
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Python lists can contain elements of any data type - strings of characters (e.g. words, sentences, etc.), whole numbers (called "integers" or ints), decimal numbers (called "floating point numbers" or floats), other lists, etc.
There are other kinds of containers in Python - dictionaries, deques, queues - each with different features and uses.  We won't look into them further here, but if you are interested, consult the documentation or an introductory text such as ThinkPython by Allen Downey.
4.2 NumPy Arrays
NumPy arrays are better containers for purely numerical data.
Can you write Python code for it?

#Create an array of 10 values between 0 and 10-1
arr = np.arange(10)

print arr

#Create an array of 10 evenly-spaced values between 0 and 10
arr2 = np.linspace(0,10,10)

print arr2


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
Runtime errors are annoying but at least they announce themselves.  What happens if we make a logical error?
Can you write Python code for it?

def cube_root(x):
    '''This function takes a number, x,
       computes the cube root,
       and returns the result
    '''
    return np.power(x,1/3)

print cube_root(8)


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
The elements of the arrays can be accessed with the same indexing and slicing as we used before:
Can you write Python code for it?

#How many elements are in the array?
print len(arr)

#Print the 5-th element (sixth in the list):
print arr[5]


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
There is extensive documentation for Python, IPython, NumPy, SciPy, Matplotlib, etc. on the web.  Chances are, your question has been asked and answered somewhere already.  Google is your friend.  A few places to look when you get stuck:

The IPython website
Stackoverflow
Reddit

5. Applications
Now that we we have seen some of the basics, let's try using what we learned to solve a simple problem.
5.1 Project Euler
Project Euler is an online repository of simple and hard math puzzles that can be solved with the computer.  These are great for practicing problem-solving, algorithmic thinking, and coding.  Let's try a simple one together.
Multiples of 3 and 5
Problem 1
If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.
Find the sum of all the multiples of 3 or 5 below 1000.
https://projecteuler.net/problem=1
Work with a partner to brainstorm a solution to this problem by breaking it into logical steps.  Type your set of steps into a markdown cell as a bulleted list.  Then try to implement the code for these steps in a code cell.  Verify your solution works for the case where N = 10 before trying N = 1000.
Steps:
*

*
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
This is an example of a syntax error.  We forgot to close the quotes around our message.  Python error messages are (usually) informative and tell you what the problem is and where it occurred.  We can fix this problem by closing our quotes.
Can you write Python code for it?


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
To execute a cell:

Shift-Enter: run cell and move to new cell below.
Control-Enter: run cell and stay in the cell.

3. Notebook basics
There are two primary different kinds of cell types - "code" and "markdown".  The cell in which this text was typed is a markdown cell.  A code cell is one in which computer instructions or code can be typed.  To execute a cell, hit Shift-Enter.
Markdown cells use the Markdown formatting system to allow you to include formatted text, such as italic and bold, or to create bulleted or numbered lists:

a list element

another list element

a numbered list element

another numbered list element

The view of these cells is determined by whether they have been executed or not.  Double-clicking any executed markdown cell will bring it into "edit" mode.  Try it with one of these cells.
Mathematical expressions can be rendered using LaTeX formatting (pronounced "Lah-Tech") to give full-featured symbolic representation:
$$x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}$$
The LaTeX "code" to create that equation is
$$x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}$$
LaTeX equation typesetting within the IPython Notebook is very helpful for displaying equations and directly connecting them with the computer code that implements them numerically.
Images can be included in markdown cells using HTML tags like

or they can be included in code cells using the display features of IPython.
Can you write Python code for it?

from IPython.display import Image
Image(filename='img/Cats.jpg')


I am working on the file "STAR2015Workshop.ipynb".
The first task is:
or designate a step size to jump over certain elements:
Can you write Python code for it?

#Print every other element:
print shopping_list[::2]

#Print every third element (skip 2):
print shopping_list[::3]


I am working on the file "STEP_2_Digest_data.ipynb".
The first task is:
------------- (semi)-Automatic -------------
Can you write Python code for it?

import digest

digest.parse(filetype, directory, data, channels, image_size)

digest.split(directory, split)

digest.class_weights(directory, data)


I am working on the file "STEP_3b_Train_prebuilt_CNN.ipynb".
The first task is:
Load data and labels:
Can you write Python code for it?

class_weights = pickle.load(open(os.path.join(input_directory, "class_weights.sav"), 'rb'))

classes = len( pickle.load(open(os.path.join(input_directory, "class_names.sav"), 'rb')) )

training_x = numpy.load(os.path.join(input_directory, "training_x.npy"))

training_y = numpy.load(os.path.join(input_directory, "training_y.npy"))

validation_x = numpy.load(os.path.join(input_directory, "validation_x.npy"))

validation_y = numpy.load(os.path.join(input_directory, "validation_y.npy"))


I am working on the file "STEP_3b_Train_prebuilt_CNN.ipynb".
The first task is:
Pre-built Keras models:
Very Deep Convolutional Networks for Large-Scale Image Recognition https://arxiv.org/abs/1409.1556
- VGG16
- VGG19
Deep Residual Learning for Image Recognition https://arxiv.org/abs/1512.03385
- ResNet50
Inception Architecture for Computer Vision https://arxiv.org/abs/1512.00567
- Inception v3
Note:
Pre-trained networks are not well-suited for bioimaging analysis of multichannel fluorescent images, i.e. they were built for 3 channels of RGB images. In contrast to our built-in CNN, here one should selectively choose maximum 3 channels that provide sufficient information for making prediction, and should omit the channels that may introduce noise.
To make it plausible for the pre-trained network’s requirement of RGB and computationally efficient, single-channel images of an object will be stacked to form an RGB image and saved in PNG format. User should be informed that several PNG files will be stored in the hard disk as the location of png_directory.
------------- (semi)-Automatic -------------
Can you write Python code for it?

import keras
import keras.preprocessing.image
import numpy
import pickle
import os.path
import tensorflow
import digest
import time


I am working on the file "STEP_3b_Train_prebuilt_CNN.ipynb".
The first task is:
Stack single-channel images (maximum 3 channels) and store to PNG files
Can you write Python code for it?

%%capture
digest.save_png(training_x, os.path.join(png_directory,"Training") )
digest.save_png(validation_x, os.path.join(png_directory,"Validation") )


I am working on the file "STEP_3b_Train_prebuilt_CNN.ipynb".
The first task is:
Train the network
Can you write Python code for it?

start = time.time()

# -------- If using Tensorflow (CPU) : -------- #

# init = tensorflow.global_variables_initializer()

# session.run(init)

# -------------------------------------------- #

model.fit_generator(
    callbacks=[
        keras.callbacks.CSVLogger( os.path.join(model_directory, 'training.csv') ),
        keras.callbacks.ModelCheckpoint( os.path.join(model_directory, 'checkpoint.hdf5') )
    ],
    epochs=18,
    class_weight = class_weights,
    verbose = 0, # ON/OFF printing output
    generator = training_generator,
    max_q_size = 256,
    steps_per_epoch=2500,
    validation_data = validation_generator,
    validation_steps = 2500
)

stop = time.time()
print(stop - start, "s")

model.save( os.path.join(model_directory, 'model.h5') )

session.close()


I am working on the file "STEP_4_Test_and_Visualization_built-in_CNN.ipynb".
The first task is:
------------- (semi)-Automatic -------------
Can you write Python code for it?

%matplotlib inline
import keras
import pickle
from keras.layers import *
from keras.models import Sequential
import numpy
import os
import os.path

import matplotlib.pyplot
import pandas
import seaborn
import sklearn.metrics
import tensorflow

from tensorflow.contrib.tensorboard.plugins import projector


I am working on the file "STEP_4_Test_and_Visualization_built-in_CNN.ipynb".
The first task is:
Configure GPU/CPU devices:
Can you write Python code for it?

# -------- If using Tensorflow-GPU: -------- #

configuration = tensorflow.ConfigProto()

configuration.gpu_options.allow_growth = True

configuration.gpu_options.visible_device_list = "0"

session = tensorflow.Session(config=configuration)

keras.backend.set_session(session)

# -------- If using Tensorflow (CPU) : -------- #

# configuration = tensorflow.ConfigProto()

# session = tensorflow.Session(config=configuration)

# keras.backend.set_session(session)

if not os.path.exists(output_directory):
    os.makedirs(output_directory)


I am working on the file "STEP_4_Test_and_Visualization_built-in_CNN.ipynb".
The first task is:
Confusion matrix
Can you write Python code for it?

predicted = model.predict(
    batch_size=50,
    x=testing_x
)

predicted = numpy.argmax(predicted, -1)
expected = numpy.argmax(testing_y[:, :], -1)

confusion = sklearn.metrics.confusion_matrix(expected, predicted)

confusion = pandas.DataFrame(confusion)

matplotlib.pyplot.figure(figsize=(12, 8))

seaborn.heatmap(confusion, annot=True)

matplotlib.pyplot.savefig( os.path.join(output_directory, 'confusion_matrix.eps') , format='eps', dpi=600)


I am working on the file "STEP_4_Test_and_Visualization_built-in_CNN.ipynb".
The first task is:
Data queueing
Can you write Python code for it?

def training_data_generator(input_x, input_y, batch_size):
    num_examples, num_labels = input_y.shape
    label_indices = []
    for i in range(num_labels):
        indices = [j for j in range(num_examples) if input_y[j,i] > 0]
        label_indices.append(indices)
        print("Label",i,":",len(indices),"examples")
    samples_per_label = int(batch_size / num_labels)

    def generator():
        while True:
            x_samples = []
            y_samples = []
            for i in range(num_labels):
                random.shuffle(label_indices[i])
                indices = label_indices[i][0:samples_per_label]
                x_samples.append( input_x[indices, ...] )
                y_samples.append( input_y[indices, ...] )
            x_samples = numpy.concatenate( x_samples )
            y_samples = numpy.concatenate( y_samples )
            batch_indices = numpy.arange(x_samples.shape[0])
            numpy.random.shuffle(batch_indices)
            x_samples = x_samples[batch_indices, ...]
            y_samples = y_samples[batch_indices, ...]
            yield (x_samples, y_samples)
    return generator()

def prediction_data_generator(input_x, input_y, batch_size):
    num_examples, num_labels = input_y.shape
    steps = int(num_examples / batch_size)
    def generator():
        i = 0
        while True:
            start = i*batch_size
            end = (i+1)*batch_size
            x_sample = input_x[start:end, ...]
            y_sample = input_y[start:end, ...]
            yield (x_sample, y_sample)
            i = i + 1 if i < steps else 0
    print("Prediction steps:",steps)
    return generator(), steps

# This function to normalize illumination discrepancy across images

def min_max_norm(x, minimum=None, maximum=None):
    channels = x.shape[-1]
    if minimum is None and maximum is None:
        minimum = []
        maximum = []
        for channel in range(channels):
            minimum.append( x[..., channel].min() )
            maximum.append( x[..., channel].max() )
    result =...


I am working on the file "STEP_4_Test_and_Visualization_built-in_CNN.ipynb".
The first task is:
Extract the most crucial layer
Can you write Python code for it?


I am working on the file "STEP_4_Test_and_Visualization_built-in_CNN.ipynb".
The first task is:
Load trained model:
(can also load checkpoints)
Can you write Python code for it?

model = keras.models.load_model( os.path.join(model_directory, 'model.h5') )

model.load_weights(os.path.join(model_directory, 'model.h5'))