I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
If you noticed above we use the == symbol to ask the computer if the numbers are equal instead of the = symbol. This is because the = symbol is used for assigning variables their values. Can you guess what the >= symbol measured?
A list of lists of things that mean 3
What if you could store more than one thing in a variable? What if you could store more than one type of thing in a variable? Well you can! Python has a type of variable called a list which can do that. There are a few python data types which you should read about, broadly you have numeric, and strings, i.e. numbers and letters. But there can also be different types of numbers and different kinds of strings.
Let's see how strings work:
Can you write Python code for it?

# assign the value 3 to 3 variables
goat = guitar = eskimo = 3
# declare and print list
my_name_is_three = [[3,3,3], ['three', 'teen', 'tres', 'san'], [goat, guitar, eskimo ]]
my_name_is_three


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
It seems to do that ok... can you make it do other operations like divide? multiply?
This is all well and good but one can't be expected to always type out numbers that you want to do operations on. There is a way to make the computer store numbers you want (or really store anything you want) and make it do operations on that.
Variables
Variables are basically bags for your items. There isn't much of a point in trying to explain variables with words let's just play with them!
Can you write Python code for it?


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
Let's break down what happened there. To do that we must go to the innermost part of the statement, the part where we are using the plus sign. We are adding the string at the first position in dir_of_root to the string in root_root. If you don't understand how that works, try doing it:
Can you write Python code for it?


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
Now we have all the filenames being accessed, but how do we make that into a full file path? Remember the os.path.join() function? That's going to help a bit
Can you write Python code for it?

for file_name in img_file_list:
    full_path = os.path.join(root_root,dir_of_root[0], file_name)
    print(full_path)


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
Ok, now we have the path to one image file.. what about the others? Our lazy butts can't seriously be expected to change the 0 to a 1 then to a 2 etc just to access every file... Again we don't have to! We can use LOOPS!
Sidenote: you might have noticed above that the filenames don't seem to be in the right order. This seems to be a property of the way python assigns priority to file reading sequence. If you'd like to know more or if you need the files to be in order you should google how to do that.
Loop! Cause I'm lazy and smart :D
Loops are what you apply once you have figured out how to do something once, but need to do it again and again by making some minor changes. In our case we need to change the file name every time. This idea of again-and-again is packaged into the word iterate. So when someone says "iterate over X" all that it means is that you apply the same things that are in X. We will be iterating over the file names stored in img_file_list.
In case you come from other languages, you can't normally iterate over objects in a container (like a list) however in python you can. Let's get our file list again:
Can you write Python code for it?

import os
root_root = '/home/aneesh/Images/Source/'
dir_of_root = os.listdir(root_root)
img_file_list = os.listdir(root_root+dir_of_root[0])


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
The variable img_file_list contains the names of all the files in the Type1 directory. For loops are used to iterate over items. The term for comes from the understanding: "for x items in y do z". This is also how we write for loops in python:
Can you write Python code for it?

for file_name in img_file_list:
    print(file_name)


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
We then passed that joined string to the os.listdir() function which gave us a list of the images in the Type1 directory. The directory names can also be joint using the os.path.join() function. This can be useful when you are dealing with many variables contributing different sources like this:
Can you write Python code for it?

full_path = os.path.join(root_root,dir_of_root[0], img_file_list[0])


I am working on the file "Notebooks/1-Your first steps .ipynb".
The first task is:
What if after storing, you wanted to only look at the first list in the list of lists? This can be done using indexing. Indexing means telling the computer to pull out a particular value at the coordinates we specify. Python uses 0-indexing. 0-indexing starts counting from 0 instead of 1. So to access the first element you have to type:
Can you write Python code for it?


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
36 Zip Codes of a more or less contiguous area is what we will be focusing on from now on.
Can you write Python code for it?

# Create a new DataFrame only including Austin City Limits restaurants:

ACL = df[df.Zip_Code.isin(ACL_zip)]
ACL.describe()


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
5. DataFrame TLC and a very interesting visualization
Our "main" DataFrame needs some TLC. We also have to exploit the "Address" column further by creating neat columns with street address, city information and coordinates for each row - even though we are focusing in the contiguous Austin area, we are still having the areas of Rollingwood, West Lake Hills, Sunset Valley and Bee Cave to take into account (and get some interesting associations between Restaurant Health Inspection Score and Income...)
Can you write Python code for it?

# Creation of a DataFrame off of the constituent components of the 'Address' Column:

merged_location = merged['Address'].apply(lambda x: pd.Series(x.split('\n')))

merged_location.head()


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
At first sight the data looks scattered all over the place for all three plots, and there's possibly no correlation between any of the pairs. However there are some interesting remarks one could make:

As we have already seen, the vast majority of restaurants scores over 70 (passing grade)
In areas (Zip Codes) where the median income is relatively lower, it is more probable that there will be restaurants doing worse than average. Notice that when the median income is higher than \$120,000, there is not a single restaurant failing the inspection. However this might just be a random fluctuation of our data, since there are a few failing restaurants in areas where the median income lies between \$80,000 and \$120,000
Population really doesn't seem to play any role; There is a more or less even distribution of health inspection grades irrespective of the Zip Code's population: both the sparsely and the densely populated areas of town have their share of passing and failing restaurants.
The scatterplot of restaurant health inspection scores vs. percentage of home ownership looks very similar to the one of scores vs median income. We can easily check whether the two quantities are correlated, which we intuitively expect.

Next: Supervised Learning - Regression Analysis
Let us save our working DataFrame as a csv file so we can import it into the next Notebook.
Can you write Python code for it?


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
Good news! It's pretty much impossible to get sick at Jamba Juice or Starbucks. The large pizza chains all do very well. Also, if you're looking for health code-friendly Asian food, Panda Express is your new best friend. If your primary basis for deciding between drive-thrus is tiny differences in health inspection score, you may want to opt for local hero Whataburger, Wendy's or Burger King over McDonald's. Otherwise, there's nothing terribly interesting here, other than the suprisingly low (for me) scores of Pei Wei, which I have to admit I used to hold in a higher esteem. An average score of 88 of course isn't something to be very disappointed about, but it seems that they, being a national chain, are slightly more lax when it comes to their food preparation protocols compared to their more "fast-food" counterpart, Panda Express.
The reader is strongly encouraged to visit Tal Yarkoni's notebook for further exploratory data analysis on the Ceity of Austin Restaurant Health Inspection dataset.
4. Enriching our dataset by adding numerics
As we have seen, our only quantitative variable in our data set is each restaurant's score. Even though the given data set can lead us to a fairly thorough exploration and raise some quite interesting questions, as we have seen so far, my opinion is that our output would be much richer if we had more information. The steps I took to address this are:

I unilaterally decided to trim down the area of interest. Health Inspections are conducted by Travis County which is rather spacious and includes the City of Austin, the neighboring communities like Pflugerville as well as some far-flung suburbs like Lago Vista or Manor. I spent some time over a map and ended up deciding to focus my analysis from now on in my loose definition of the Austin City Limits. Roughly this translates as eliminating all rows from my DataFrames that belong to the suburban Zip Codes.

I did some more research on the remaining 36 Zip Codes after the purge above, and following U.S. Census Data from 2014 I...
Can you write Python code for it?

# Let's see how many Zip Codes do we start with:

print len(pd.unique(df.Zip_Code.ravel()))


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
Let me now filter out everything that doesn't follow my loose definition of a restaurant:
Can you write Python code for it?

df = df[df['Restaurant_Name'].str.contains('School') == False]
df = df[df['Restaurant_Name'].str.contains('Elementary') == False]
df = df[df['Restaurant_Name'].str.contains('Care') == False]
df = df[df['Restaurant_Name'].str.contains('Middle') == False]
df = df[df['Restaurant_Name'].str.contains('Cafeteria') == False]
df = df[df['Restaurant_Name'].str.contains('Jail') == False]
df = df[df['Restaurant_Name'].str.contains('ISD') == False]
df = df[df['Restaurant_Name'].str.contains('Academy') == False]

df = df[df['Restaurant_Name'].str.contains('Mart') == False]
df = df[df['Restaurant_Name'].str.contains('Gas') == False]
df = df[df['Restaurant_Name'].str.contains('Convenience') == False]
df = df[df['Restaurant_Name'].str.contains('7-Eleven') == False]
df = df[df['Restaurant_Name'].str.contains('HEB') == False]
df = df[df['Restaurant_Name'].str.contains('Station') == False]
df = df[df['Restaurant_Name'].str.contains('Randall') == False]
df = df[df['Restaurant_Name'].str.contains('Target') == False]
df = df[df['Restaurant_Name'].str.contains('Flea') == False]

df = df[df['Restaurant_Name'].str.contains('Gym') == False]
df = df[df['Restaurant_Name'].str.contains('Fitness') == False]
df = df[df['Restaurant_Name'].str.contains('Fit') == False]

df = df[df['Restaurant_Name'].str.contains('Church') == False]
df = df[df['Restaurant_Name'].str.contains('Dollar') == False]
df = df[df['Restaurant_Name'].str.contains('Store') == False]
df = df[df['Restaurant_Name'].str.contains('Texaco') == False]

print len(df)

restaurants = df.groupby('Facility_ID') # We can switch from "establishments" to "restaurants" after the purge
len(restaurants)


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
Let us also create an auxiliary DataFrame where each establishment shows up once (even though there are 5-6 inspection instances per establishment) - we need to know how many individual establishments are inspected. Side note: I am using the word "establishment" instead of "restaurant", because the Austin/Travis County Health and Human Services Department conducts the permitting and inspection of more than 4,000 food establishments in Austin, several local municipalities and rural Travis County. This includes any place serving food: restaurants, school cafeterias, grocery and convenience stores etc. We will deal with this later, because we have chosen to focus our study on restaurants (loosely defined as places that a potential patron chooses to go dine or carry out)
Can you write Python code for it?

establishments = df.groupby('Facility_ID')

# Print some stuff
print "Number of inspections:", len(df)
print "Number of individual establishments inspected:", len(establishments)
print "\nColumns:\n", df.dtypes


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
Let us now introduce the U.S. 2014 Census numerics data:
Can you write Python code for it?


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
The "Inspection_Date" column looks like a humanly readable series of dates, however it is encoded as (useless) strings. Fortunately we can easily tell pandas to see this column as a datetime format for easier and better reference:
Can you write Python code for it?

from datetime import datetime

df['Inspection_Date'] = pd.to_datetime(df['Inspection_Date'])


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
The first striking result is that, given the health inspectors are doing their job meticulously, Austin is a quite sanitary city for diners. Recall that a failing score is below 70, and according to the ad hoc letter grade scheme we are going to implement, the vast majority of inspections leeds to a letter grade "A", which we would say symbolizes a "pristine" establishment. We are going to perform some thorough research in the Machine Learning part of this Project, however this dataset can yield some quite interesting visualizations:
Can you write Python code for it?

# Function to plot a histogram based on the frequency table above:

def plot_histogram():
    df['Score'].hist(bins=50)
    plt.xlabel('Health Inspection Score')
    plt.ylabel('Count')
    plt.savefig('histogram.png')

plot_histogram()

# A somewhat prettier histogram with seaborn; Relative frequencies on y-axis
sns.distplot(df['Score']);


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
There is a sharp break at 70. Recall that 70 is the minimum score required to pass inspection. If a restaurant gets a score below that, it fails inspection, which presumably triggers all kinds of unpleasant things - e.g., more paperwork for the inspector, bad PR for the restaurant, and a follow-up (non-routine) inspection one or two weeks later. So one possibility here is that there's some pressure on inspectors - whether explicit or implicit - to avoid giving restaurants failing scores. This could possibly explain the fact that 99% of the entries in our data set have passing scores, as we are going to see later. Unfortunately this great imbalance between passing and failing restaurants will significantly hinder our attempts to build predictive models with confidence.
Dr. Tal Yarkoni offers some more interesting explanations: For example, it's possible that the health inspection guidelines explicitly allow inspectors to let restaurants off with a warning. Or perhaps the scoring algorithm isn't just a linear sum of all the violations observed, and there has to be some particularly egregious health code violation in order for a restaurant to receive a score below 70. Or, inspectors may be pragmatically working around weaknesses in the code - e.g., a restaurant may theoretically be able to fail inspection because of a large number of minor infractions, even if no single infraction presents any meaningful health risk to customers.
Still, absent an explanation, there's at least some potential cause for concern here, and it's certainly consistent with the data that health inspectors might be systematically grading restaurants more leniently than they should be. Are the inspectors doing their job meticulously? Is Austin dining really so sanitary overall?
This is a recurring theme of this study, as we are going to see.
2. Data Preparation
On to the immensely boring, yet quintessential part: data cleaning. I will start by renaming some of our restaurants so they'll avoid the axe when I remove everything which isn't a...
Can you write Python code for it?

# Unfortunately the words "Market", "Grocery", "Church" correspond to places
# like convenience stores and grocery stores.

# I want to make sure that some restaurants with those words in their name will go through my filter.

df.loc[(df['Restaurant_Name'].str.contains('Whole')) , 'Restaurant_Name'] = "Whole Foods"
df.loc[(df['Restaurant_Name'].str.contains('Central Market')) , 'Restaurant_Name'] = "Central Mkt"
df.loc[(df['Restaurant_Name'].str.contains('Boston Market')) , 'Restaurant_Name'] = "Boston Mkt"
df.loc[(df['Restaurant_Name'].str.contains('Mandola')) , 'Restaurant_Name'] = "Mandola's"
df.loc[(df['Restaurant_Name'].str.contains('Royal Blue')) , 'Restaurant_Name'] = "Royal Blue"
df.loc[(df['Restaurant_Name'].str.contains('Rudy')) , 'Restaurant_Name'] = "Rudy's"
df.loc[(df['Restaurant_Name'].str.contains('Fit Foods')) , 'Restaurant_Name'] = "My Ft Foods"
df.loc[(df['Restaurant_Name'].str.contains("Church's Chicken")) , 'Restaurant_Name'] = "Chrch Chicken"
df.loc[(df['Restaurant_Name'].str.contains("Schlotzsky's")) , 'Restaurant_Name'] = "Schlotzsky's"

len(df)


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
Those 56 Zip Codes cover the entire area of Travis County where the Austin/Travis County Health and Human Services Department conducts the inspections. After spending some time studying a detailed map, I decided to only keep a subset of those 56 Zip Codes from now on, even though this means I am trimming down my data set. I am interested in Austin proper:
Can you write Python code for it?


I am working on the file "Notebooks/1. Data Preparation & Exploratory Analysis.ipynb".
The first task is:
We have nearly 23,000 inspections from over 4,700 distinct establishments. For each inspection, we have (among other things) a health inspection score, an establishment name, an inspection date, and a description of the process (i.e., whether the inspection was routine, or reflected a follow-up visit triggered by an establishment failing an earlier inspection).
Let us start our exploratory data analysis by a simple frequency table of the health inspection scores:
Can you write Python code for it?

print df['Score'].value_counts().sort_index() # Frequency Table


I am working on the file "Notebooks/1.0_Explicito.ipynb".
The first task is:
La funcion solve_explicito nos devuelve dos cosas. Una son los array de temperatura para distintos tiempo, y lo otro es un diccionario de los valores usados para el calculo de la solución numérica.
Can you write Python code for it?


I am working on the file "Notebooks/1.0_Explicito.ipynb".
The first task is:
Observar que en esta ultima simulación, logramos bajar 2 ordenes de magnitud en $\Delta z$ (de 1800 a 10) y un orden de magnitud en $\Delta t$ (de $10^{8}$ a $10^{7}$).
Siguiente paso: Método Implicito
The cell below loads the style of the notebook.
Can you write Python code for it?

#Este asombroso css es trabajo de @LorenaABarba
from IPython.core.display import HTML
css_file = './style/css/IPython_personal_style.css'
HTML(open(css_file, "r").read())


I am working on the file "Notebooks/1.1_Probando_Cython.ipynb".
The first task is:
Copiamos la implementación de python en un archivo separado que termina en .pyx y lo compilamos usando el scrip setup.py haciendo:
$ python setup.py build_ext --inplace
Can you write Python code for it?

a = time.time()
t_out, dic = solve_explicit(metodo='explicit_cython')
print('Explicito cython demoro ',time.time() - a, 'segundos')
pretty_plot(t_out, dic)


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Dan nilai drawdawn terbesar dapat kita lihat sebagai berikut:
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Dari grafik di atas kita bisa melihat maximum drawdawn terjadi sekitar tahun 1929, atau di saat the great recession. Kita bisa melihat waktu tepatnya kapan sebagai berikut:
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Dari grafik di atas terlihat jelas bahwa karakter pergerakan nilai portfolio kedua aset sangat berbeda, dan hasil akhirnya (setelah 12 bulan) juga sedikit berbeda (aset biru sedikit mengungguli aset oranye). Kedua hal ini tidak tertangkap dengan hanya memakai nilai rata-rata keuntungan (ingat, rata-rata keuntungan keduanya sama).
Keuntungan Majemuk (Compound Return)
Misalkan $R_{t0,t1}$ adalah keuntungan yang diperoleh di perioda t0 ke t1, dan $R_{t1,t2}$ adalah keuntungan yang diperoleh di perioda t1 ke t2, maka $R_{t0,t2}$ yaitu keuntungan yang diperoleh di total perioda dari t0 ke t2 adalah:
$$ R_{t0,t2} = (1 + R_{t0,t1}) * (1 + R_{t1,t2}) - 1 $$
Menyetahunkan Keuntungan (Annualizing Return)
Seringkali kita ingin membandingkan keuntungan yang didapat dari perioda yang berbeda. Misalnya antara keuntungan bulanan dengan keuntungan per kuartal. Agar bisa dibandingkan, biasanya kedua keuntungan tersebut kita setahunkan (annualize) dahulu.
Formulanya adalah sebagai berikut.
$$R_{tahunan} = (1 + R_{perioda}) ^{N_p} - 1$$
di mana $N_p$ adalah jumlah perioda dalam setahun.
Misalnya, untuk menyetahunkan keuntungan bulanan:
$$R_{tahunan} = (1 + R_{bulanan}) ^{12} - 1$$
Untuk menyetahunkan keuntungan kuartalan:
$$R_{tahunan} = (1 + R_{kuartalan}) ^{4} - 1$$
Karakterisasi 2: Volatilitas
Kita sudah sedikit menyinggung tentang volatilitas di atas. Di bagian ini kita akan membahasnya dengan lebih detil.
Ukuran pertama untuk menggambarkan volatilitas adalah variance. Variance, dilambangkan  σ2 (sigma kuadrat), dihitung dengan formula berikut:
$$ \sigma^2 = \frac{1}{N} \sum_{i=1}^ {N} (R_i - \overline{R} )^2 $$
dimana:

$\sigma^2$ : variance
$N$ : jumlah periode (misalnya 12 untuk contoh kita di atas)
$\overline{R}$ : keuntungan rata-rata

Disamping variance, ukuran yang lebih sering digunakan adalah standard deviasi, yang dilambangkan dengan σ (sigma), notabene adalah akar dari variance:
$$ \sigma = \sqrt{ \frac{1}{N} \sum_{i=1}^ {N} (R_i - \overline{R} )^2 } $$
Standard deviasi lebih sering dipergunakan dari...
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Dari grafik di atas, secara intuitif kita bisa melihat keduanya berbeda. Misalnya, aset Oranye mempunyai volatilitas yang lebih besar dari aset Biru. Mari kita lihat karakterisasi kedua aset itu secara lebih detil.
Karakterisasi 1: Rata-Rata Keuntungan (Average Returns)
Tahukah Anda, bahwa  kedua aset tersebut menghasilkan keuntungan rata-rata yang sama tiap bulannya?
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Dari kedua nilai baik variance maupun standard deviasi di atas kita bisa lihat bahwa volatilitas dari aset Oranye lebih besar dari aset Biru. Artinya keuntungan dari aset Oranye lebih sering menyimpang jauh dari nilai rata-ratanya.
Menyetahunkan Volatilitas
Sama seperti keuntungan, kadang kita ingin membandingkan volatilitas dari dua periode yang berbeda. Agar keduanya dapat dibandingkan, maka kita bisa menyetahunkan kedua volatilitas tersebut terlebih dahulu. Formulanya adalah sebagai berikut:
$$ \sigma_{ann} = \sigma_p \sqrt{N_p} $$
dimana:

$\sigma_{ann}$: volatilitas yang disetahunkan (annualized volatility)
$\sigma_p$: volatilitas dari suatu periode p
$N_p$: jumlah periode dalam setahun. Misalkan p adalah bulanan, maka $N_p$ adalah 12 (ada 12 bulan dalam 1 tahun), dan misalnya p adalah harian, maka $N_p$ adalah 243 (rata-rata ada 243 hari kerja bursa IDX dalam setahun, dalam 10 tahun terakhir).

Sebagai contoh, mari kita setahunkan volatilitas bulanan dari contoh kita di atas.
Can you write Python code for it?

annualized_vol = returns.std() * (12**0.5)
annualized_vol


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Drawdown dihitung dengan menghitung keuntungan investasi pada suatu saat dibanding dengan puncak sebelumnya:
Can you write Python code for it?

drawdown = (wealth_index - previous_peaks) / previous_peaks
drawdown.plot()


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Karakterisasi 3: Risk Adjusted Returns
Risk adjusted return adalah salah satu cara untuk membandingkan dua aset yang mempunyai keuntungan dan volatilitas yang berbeda. Sebagai demonstrasi,
Untuk mendemonstrasikan bagian ini, kita akan memuat sebuah dataset baru yang berisi keuntungan bulanan dari dua jenis aset di Dow Jones, yaitu SmallCap dan BigCap mulai dari tahun 1926 sampai 2018.
Can you write Python code for it?

returns = pd.read_csv('SmallLargeCapReturns.csv', index_col='date', parse_dates=True)
returns.index = returns.index.to_period('M')
returns.head()


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Kedua aset tersebut sama-sama memberikan rata-rata keuntungan sebesar 1% tiap bulannya.
Dengan demikian, kita dapat simpulkan bahwa rata-rata keuntungan (average returns) bukanlah cara yang baik untuk melihat karakteristik suatu aset, bahkan mungkin bisa menyesatkan.
Cara lain untuk melihat perbedaan adalah dengan melihat apa yang terjadi kalau kita berinvestasi pada kedua aset di atas. Misalkan kita tanamkan uang satu juta rupiah, maka berikut adalah hasilnya setelah 12 bulan.
Can you write Python code for it?

(returns + 1).cumprod().plot(figsize=FIGSIZE, grid=True)


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Kedua jenis aset mempunyai rata-rata dan standard deviasi keuntungan yang berbeda.
Untuk menghitung standard deviasi, pertama kita hitung standard deviasi dari keuntungan bulanan, yaitu:
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Kemudian kita setahunkan nilai return bulanan di atas:
Can you write Python code for it?

ann_returns = ((1 + return_per_month) ** 12) - 1
ann_returns


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Kita bisa zoom pergerakan nilai investasi di sekitar perioda the great recession dengan cara berikut:
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Kita plot data keuntungan tersebut:
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Lalu kita kalkulasi nilai puncak yg tercapai sebelumnya.
Can you write Python code for it?

previous_peaks = wealth_index.cummax()
previous_peaks.plot()


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Sampai di sini, kita sudah mendapat data-data keuntungan dan standard deviasi setahun dari kedua aset:
- aset SmallCap mempunyai rata-rata keuntungan disetahunkan 16.75% dengan standard deviasi 10.63%
- aset LargeCap mempunyai rata-rata keuntungan disetahunkan 9.28% dengan standard deviasi 5.39%
Untuk menilai keuntungan aset mana yang lebih baik, salah satu cara sederhana adalah dengan membagi return dengan risk, sbb:
$$ risk\ adjusted\ return = \frac{return}{risk}$$
Rasio di atas juga disebut return to risk ratio.
Definisi dari risk adjusted return adalah berapa besar return yang dihasilkan untuk setiap unit risk.
Dalam hal ini, kita menggunakan standard deviasi sebagai ukuran risk. Dengan demikian, maka untuk menghitung risk adjusted return, kita membagi return dengan standard deviasi, sbb:
Can you write Python code for it?

risk_adjusted_returns = ann_returns / annualized_vol
risk_adjusted_returns


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10011 - Return Analysis.ipynb".
The first task is:
Setelah kita dapatkan total return, maka kita hitung return bulanannya:
Can you write Python code for it?

N_MONTHS = len(returns)
return_per_month = ((1 + total_returns) ** (1 / N_MONTHS)) - 1
return_per_month


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Cukup dekat.
Selanjutnya kita akan gambarkan return bulanan yang kita sintesa di atas. Dalam grafik di bawah, kita gambarkan juga garis oranye yang mencerminkan standard deviasi +3 dan -3, yang mencerminkan 0.9973 dan 0.0027 quantile dari data kita. Artinya, hanya ada kemungkinan 0.54% (=0.0027*2) bahwa sample-nya di atas atau di bawah garis ini. Dalam konteks return bulanan, kejadian ini hanya akan terjadi setiap 15 tahun (=1/0.0054).
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(15,6))
x = range(N)
ax.bar(x, norm_returns, label='Monthly Return')
ax.axhline(mean + 3 * std, color='C1', label='0.997 quantile')
ax.axhline(mean - 3 * std, color='C1', label='0.003 quantile')
ax.set_title('Normally Distributed Returns (Simulated)')
ax.legend()


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Distribusi Normal
Sekarang mari kita mensintesis data return bulanan dengan distribusi Normal, dengan menggunakan mean dan standard deviation (std) dari data return dari saham SmallCap.
Kita hitung mean dan standard deviasi-nya sebagai berikut:
Can you write Python code for it?

N = len(returns)
mean = returns['SmallCap'].mean()
std = returns['SmallCap'].std()
print('N: {}, mean: {}, std: {}'.format(N, mean, std))


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Histogram di atas bukanlah distribusi Normal, karena tidak simetris dan juga mempunyai ekor yang panjang (fat tail). Kedua karakteristik ini disebut skewness dan kurtosis.
Skewness dan Kurtosis
Dari studi di atas, tampaknya kita harus mempelajari lebih jauh dari sekedar melihat mean dan variance. Kita harus melihat higher order moments, dalam hal ini terutama adalah skewness dan kurtosis.
Skewness adalah ukuran dari simetri sebuah distribusi. Kita tahu bahwa distribusi normal adalah simetris; probabilitas untuk mendapatkan suatu nilai di atas dan di bawah mean (misalnya +1% dan -1% di atas/bawah mean) adalah sama. Jika distribusinya mempunyai skew yang negatif, artinya kemungkinan untuk mendapatkan sampel di bawah mean lebih besar dari pada kemungkinan mendapatkan sampel di atas mean. Distribusi Normal mempunyai nilai skewness nol.
Kurtosis adalah ukulan ketebalan dari ekor distribusi. Kita tahu bahwa distribusi Normal mempunyai ekor yang sangat tipis. Distribusi probabilitasnya turun secara drastis menuju nol. Distribusi Normal mempunyai nilai kurtosis tiga. Nilai kurtosis lebih besar dari tiga berarti distribusi tersebut mempunyai ekor yang tebal (fat tail).
Excess kurtosis adalah nilai kurtosis sebuah distribusi dikurangi tiga. Untuk distribusi normal, excess kurtosis bernilai nol.
Menghitung Skewness dan Kurtosis dengan Python
Di Python sudah tersedia fungsi untuk menghitung skewness dan excess kurtosis.
Mari kita gunakan pertama untuk data buatan yang seharusnya berdistribusi Normal.
Can you write Python code for it?

from scipy.stats import kurtosis, skew

sk = skew(norm_returns)
kr = kurtosis(norm_returns)
print('Skewness: {}, excess kurtosis: {}'.format(sk, kr))


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Kita bisa verifikasi mean dan standard deviasinya sesuai (catatan: ketidaksesuaian mean dan std dari random samples ini biasanya disebabkan karena jumlah samples yang kita sintesis tidak banyak. Semakin besar N, maka mean dan std akan semakin mendekati parameternya):
Can you write Python code for it?

print('Mean: {}, std: {}'.format(norm_returns.mean(), norm_returns.std()))


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Kita juga bisa plot distribusinya sebagai histogram:
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Maka tampak histogram yang bentuknya mirip dengan distribusi normal. Kalaupun ada perbedaan, ini dikarenakan jumlah sampel yang kita sintesis tidak cukup banyak.
Distribusi dari Return Aktual
Dengan mekanisme yang sama, mari kita gambarkan return yang sebenarnya dari aset SmallCap.
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(15,6))
x = range(len(returns))
ax.bar(x, returns['SmallCap'], label='Monthly returns')
ax.axhline(mean + 3 * std, color='C1', label='0.997 quantile')
ax.axhline(mean - 3 * std, color='C1', label='0.003 quantile')
ax.set_title('Actual Monthly Returns')
ax.legend()


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Seharusnya kedua nilai di atas adalah nol, tapi sekali lagi hal ini disebabkan karena jumlah sampel kita cukup sedikit.
Sekarang mari kita hitung untuk return dari SmallCap.
Can you write Python code for it?

sk = skew(returns['SmallCap'])
kr = kurtosis(returns['SmallCap'])
print('Skewness: {}, excess kurtosis: {}'.format(sk, kr))


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Seperti terlihat, angka-angkanya jauh menyimpang dari nol, sehingga dapat disimpulkan distribusinya bukan Normal.
Tes Normalitas dengan Jargue Bera
Tes Jargue Bera dapat digunakan untuk mentes apakah suatu sampel data mempunyai skewness dan kurtosis yang cocok dengan distribusi Normal.
Tes ini juga sudah diimplementasikan di Python. Catatan: Jargue-Bera test membutuhkan data paling tidak 2000 sampel, sedangkan data kita hanya 1100, jadi hasilnya mungkin tidak sepenuhnya benar.
Can you write Python code for it?

from scipy.stats import jarque_bera

_, pval = jarque_bera(norm_returns)
print('Distribusinya {} (pval: {:.2f})'.format('normal' if pval >= 0.05 else 'bukan normal', pval))

_, pval = jarque_bera(returns['SmallCap'])
print('Distribusinya {} (pval: {:.2f})'.format('normal' if pval >= 0.05 else 'bukan normal', pval))


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Setelah itu kita bisa mensintesis data buatan dengan distribusi Normal dengan kedua parameter di atas, lalu kita simpan sebagai Pandas Series agar mudah di-plot, sebagai berikut.
Can you write Python code for it?

np.random.seed(11)
norm_returns = pd.Series(np.random.normal(mean, std, N))


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Terlihat bahwa return yang aktual lebih banyak melampaui batas +/- 3 SD. Kita bisa menghitung berapa kali dia melampai batas dengan kode berikut.
Can you write Python code for it?

plus3sd = returns['SmallCap'].mean() + 3 * returns['SmallCap'].std()
minus3sd = returns['SmallCap'].mean() - 3 * returns['SmallCap'].std()
print('+3 SD: {}, -3 SD: {}'.format(plus3sd, minus3sd))

outside = returns[(returns['SmallCap'] > plus3sd) | (returns['SmallCap'] < minus3sd)]
outside['SmallCap']


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10012 - Return Distribution Analysis.ipynb".
The first task is:
Terlihat di atas bahwa return bulanan berapa di luar batas +/- 3 SD hanya dua kali.
Kalau kita plot distribusinya sebagai histogram:
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10013 - VaR (Value at Risk) dan CVaR (Conditional Value at Risk).ipynb".
The first task is:
Value at Risk (VaR)
Value at Risk (VaR) mengukur maksimum kerugian yang bisa dialami dari suatu investasi setelah kasus kerugian ekstrim dihilangkan. Jadi 95% Value at Risk berarti maksimum kerugian yang bisa dialami setelah 5% kerugian yang paling ekstrim dihilangkan.
Angka 95% ini, disebut confidence level, dan bisa diganti dengan yang lain sesuai kebutuhan, misalnya 99% atau 99.9%.
VaR bisa dipakai oleh manajer investasi untuk menyisihkan cadangan modal yang dipakai untuk menutup kemungkinan kerugian terburuk yang bisa dialami.
Untuk menghitung VaR, pertama tentukan batas confidence level-nya, misalkan 95%. Karena batasnya 95%, maka buang 5% data return yang terendah. Lalu ambil return terendah dari data return yang tersisa, lalu  dijadikan positif. Itulah nilai VaR-nya.
Untuk menghitung 95% VaR dari data di atas, pertama kita hilangkan 5% data return terburuk, atau dalam hal ini berarti kita hilangkan lima sampel. Nilai VaR-nya adalah nilai terburuk yang tersisa (-0.16138978) lalu kita positifkan (menjadi 0.16138978).
Artinya, dalam 95% kemungkinan, kerugian terburuk investasi kita adalah minus 0.16138978 atau minus 16%.
Cara lain untuk menghitung VaR adalah dengan memanggil fungsi np.percentile() seperti di bawah ini.
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Dan juga volatilitasnya:
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Done! Kita telah menggambar efficient frontier untuk 2 aset.
Praktek Efficient Frontier untuk Beberapa Aset
Garis besar langkah-langkahnya kurang lebih adalah sebagai berikut:
- kita tahu return terkecil dari portfolio adalah kalau 100% alokasi diberikan kepada aset dengan return terkecil, dan return terbesar dari portfolio adalah kalau 100% alokasi diberikan kepada aset dengan return terbesar. Maka return terkecil dan terbesar ini adalah titik awal dan akhir dari kurva EF kita (return terkecil adalah ujung bawah kurva, return terbesar adalah ujung atas kurva). Contoh:
  - misalnya return terkecil diberikan oleh aset A yaitu sebesar 10% dan return terbesar diberikan oleh aset B yaitu 90%.
- lalu kita bagi range dari return terendah-tertinggi di atas menjadi beberapa bagian (n_points). Contoh:
  - misalnya kita bagi menjadi 20 bagian (n_points=20), maka kita akan mempunyai list dengan isi 20 element, mulai dari return terendah yaitu 10% sampai tertinggi yaitu 90%: [10.0, 14.2, 18.4, ..., 85.8, 90.0]
- tiap-tiap bagian merepresentasikan suatu return tertentu. Untuk setiap return ini, kita akan mencari alokasi (weights) yang tepat yang dapat memberikan volatilitas yang terendah. Proses ini dilakukan di fungsi minimize_vol() di bawah. Fungsi ini memanggil fungsi minimize() yang disediakan oleh pustaka scipy.optimize.
- pada akhir iterasi, kita mendapatkan kombinasi weights yang tepat yang memberikan volatilitas terendah untuk setiap return pada slice yang kita tentukan di atas.
- dengan weights itu, kita bisa hitung volatilitas dari portfolio dengan memanggil portfolio_vol(), dan juga returnnya dengan memanggil portfolio_return().
- dengan return dan volatilitas di tangan, kita bisa membuat grafik efficient frontier.
Kali ini langsung akan saya copy-paste kodenya dari lab MOOC Introduction to Portfolio Construction and Analysis with Python minggu kedua.
Can you write Python code for it?

from scipy.optimize import minimize

def minimize_vol(target_return, er, cov):
    """
    Returns the optimal weights that achieve the target return
    given a set of expected returns and a covariance matrix
    """
    n = er.shape[0]
    init_guess = np.repeat(1/n, n)
    bounds = ((0.0, 1.0),) * n # an N-tuple of 2-tuples!
    # construct the constraints
    weights_sum_to_1 = {'type': 'eq',
                        'fun': lambda weights: np.sum(weights) - 1
    }
    return_is_target = {'type': 'eq',
                        'args': (er,),
                        'fun': lambda weights, er: target_return - portfolio_return(weights,er)
    }
    weights = minimize(portfolio_vol, init_guess,
                       args=(cov,), method='SLSQP',
                       options={'disp': False},
                       constraints=(weights_sum_to_1,return_is_target),
                       bounds=bounds)
    return weights.x

def optimal_weights(n_points, er, cov):
    """
    """
    target_rs = np.linspace(er.min(), er.max(), n_points)
    weights = [minimize_vol(target_return, er, cov) for target_return in target_rs]
    return weights

def plot_ef(n_points, er, cov):
    """
    Plots the multi-asset efficient frontier
    """
    weights = optimal_weights(n_points, er, cov)
    rets = [portfolio_return(w, er) for w in weights]
    vols = [portfolio_vol(w, cov) for w in weights]
    ef = pd.DataFrame({
        "Returns": rets,
        "Volatility": vols
    })
    ax = ef.plot.line(x="Volatility", y="Returns", style='.-',
                      label='Efficient Frontier', legend=True, figsize=(15,6))
    ax.set_ylabel('Returns')
    ax.set_xlim(left=0)
    ax.set_ylim(bottom=0)
    return ax

assets = ['Smoke', 'Fin', 'Games', 'Coal']
plot_ef(25, er[assets], cov.loc[assets, assets])


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Jadi alokasi yang optimal adalah Food 75% dan Steel 25%.
Sekarang mari kita lihat perubahan pada alokasi aset kalau expected return dari asetnya kita rubah sedikit.
Can you write Python code for it?

msr(RISKFREE_RATE, np.array([0.11, 0.12]), cov.loc[assets, assets])


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Kelemahan Efficient Frontier
Walaupun pada awalnya Efficient frontier terlihat sangat menjanjikan, ternyata kemudian diketahui bahwa pendekatan ini mempunyai kelemahan yang cukup fatal, yang bahkan membuatnya kurang layak (feasible) untuk dipakai. Kelemahannya adalah, EF membutuhkan nilai expected return yang akurat, dan sedikit perbedaan pada nilai expected return akan menyebabkan perubahan yang drastis pada alokasi terhadap aset-asetnya.
Yuk kita demonstrasikan berikut. Agar sederhanan kita pakai dua aset saja.
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Kita bisa hitung expected return dari portfolionya dengan alokasi ini:
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Kita buat covariance matrix-nya.
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Kita lakukan hal yang sama untuk volatility.
Can you write Python code for it?

vols = [portfolio_vol(w, cov.loc[assets,assets]) for w in weights]
vols


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Maka prosentase alokasinya sudah berubah jauh. Padahal perubahannya tidak sampai 1%.
Sekarang coba kita rubah expected return-nya lebih banyak.
Can you write Python code for it?

msr(RISKFREE_RATE, np.array([0.10, 0.13]), cov.loc[assets, assets])


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Mari kita gabungkan return dan volatilitas menjadi dataframe agar mudah diplot.
Can you write Python code for it?

ef = pd.DataFrame(data={'Return': rets,
                        'Volatility': vols})
ef


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Mari membuat fungsi untuk menghitung expected return dan volatilitas dari portfolio dengan operasi matriks seperti dijelaskan di atas.
Can you write Python code for it?

def portfolio_return(weights, returns):
    return weights.T @ returns

def portfolio_vol(weights, covmat):
    return (weights.T @ covmat @ weights)**0.5


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Misalkan 4 aset tadi kita alokasikan secara seimbang:
Can you write Python code for it?


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Sekarang mari kita hitung returns dari kombinasi alokasi di atas, dan kita masukkan dalam list.
Can you write Python code for it?

rets = [portfolio_return(w, er[assets]) for w in weights]
rets


I am working on the file "Notebooks/100 - Portfolio Construction and Analysis/10021 - Efficient Frontier, Capital Market Line.ipynb".
The first task is:
Sekarang mari kita plot volatility vs return-nya.
Can you write Python code for it?

ef.plot.line(x='Volatility', y='Return',
             title='Efficient Frontier dengan Dua Aset ({} dan {})'.format(assets[0], assets[1]),
             figsize=(15,6), style='.-')


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Bad cuts
Can you write Python code for it?

asseras_bad_cuts = iu.open_many_samples( \
                  ["not_code/words/bad_cuts/asseras/1.png",
                   "not_code/words/bad_cuts/asseras/2.png",
                   "not_code/words/bad_cuts/asseras/3.png",
                   "not_code/words/bad_cuts/asseras/4.png",
                   "not_code/words/bad_cuts/asseras/5.png"])
(asseras_bad_cuts, _) = kiu.adjust_input(np.asarray(asseras_bad_cuts))

u.plot_some_images(asseras_bad_cuts, (img_cols, img_rows), grid_x=5, grid_y=1)


beneficiu_bad_cuts = iu.open_many_samples( \
                    ["not_code/words/bad_cuts/beneficiu/1.png",
                     "not_code/words/bad_cuts/beneficiu/2.png",
                     "not_code/words/bad_cuts/beneficiu/3.png",
                     "not_code/words/bad_cuts/beneficiu/4.png",
                     "not_code/words/bad_cuts/beneficiu/5.png",
                     "not_code/words/bad_cuts/beneficiu/6.png",
                     "not_code/words/bad_cuts/beneficiu/7.png",
                     "not_code/words/bad_cuts/beneficiu/8.png"])
(beneficiu_bad_cuts, _) = kiu.adjust_input(np.asarray(beneficiu_bad_cuts))

u.plot_some_images(beneficiu_bad_cuts, (img_cols, img_rows), grid_x=4, grid_y=2)


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Constants
Can you write Python code for it?

# input image dimensions
img_rows, img_cols = 34, 56

# number of networks for ensamble learning
number_of_models = 5

# checkpoints dir
checkpoints_dir = "checkpoints"

# size of pooling area for max pooling
pool_size1 = (2, 2)
pool_size2 = (3, 3)
# convolution kernel size
kernel_size1 = (4, 4)
kernel_size2 = (5, 5)
# dropout rate
dropout = 0.15
# activation
activation = 'relu'


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Experiment 1 ("asseras")
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Helper functions for model definition and load
Can you write Python code for it?

def initialize_network_single_column(model, nb_classes, nb_filters1, nb_filters2, dense_layer_size1):
    model.add(Convolution2D(nb_filters1, kernel_size1[0], kernel_size1[1],
                            border_mode='valid',
                            input_shape=input_shape, name='covolution_1_' + str(nb_filters1) + '_filters'))
    model.add(Activation(activation, name='activation_1_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size1, name='max_pooling_1_' + str(pool_size1) + '_pool_size'))
    model.add(Convolution2D(nb_filters2, kernel_size2[0], kernel_size2[1]))
    model.add(Activation(activation, name='activation_2_' + activation))
    model.add(MaxPooling2D(pool_size=pool_size2, name='max_pooling_1_' + str(pool_size2) + '_pool_size'))
    model.add(Dropout(dropout))

    model.add(Flatten())
    model.add(Dense(dense_layer_size1, name='fully_connected_1_' + str(dense_layer_size1) + '_neurons'))
    model.add(Activation(activation, name='activation_3_' + activation))
    model.add(Dropout(dropout))
    model.add(Dense(nb_classes, name='output_' + str(nb_classes) + '_neurons'))
    model.add(Activation('softmax', name='softmax'))

    model.compile(loss='categorical_crossentropy',
                  optimizer='adadelta',
                  metrics=['accuracy', 'precision', 'recall'])

def try_load_checkpoints(model, checkpoints_filepath, warn=True):
    # loading weights from checkpoints
    if os.path.exists(checkpoints_filepath):
        model.load_weights(checkpoints_filepath)
    elif warn:
        print('Warning: ' + checkpoints_filepath + ' could not be loaded')

def initialize_network_multi_column(merged_model, models):
    merged_model.add(Merge(models, mode='ave'))
    merged_model.compile(loss='categorical_crossentropy',
                         optimizer='adadelta',
                         metrics=['accuracy', 'precision', 'recall'])

def create_and_load_network(number_of_models, checkpoint_paths, nb_classes,
                            nb_filters1, nb_filters2,...


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Helper functions for prediction
Can you write Python code for it?

def predict_pipeline1(data, count_letter=True):
    start_time = time.time()
    count = 0
    for bad_cut in data:
        flag = False
        count += 1
        bad_cuts = np.asarray([bad_cut])
        if count_letter:
            print ("Predictions for the supposed letter number " + str(count))
        for char in ALPHABET_ALL:
            predictions = single_letter_models[char].predict([bad_cuts] * number_of_models)
            if (predictions[0][1] > predictions[0][0]):
                print ("Cut " + str(count) + " has been classified as good corresponding to char '" +\
                       char + "' with a confidence of " + str(predictions[0][1] * 100) + "%")
                flag = True
        if not flag:
            print ("Bad cut")
        print ("---")
    elapsed_time = time.time() - start_time
    print("Elapsed time:", elapsed_time)

def predict_pipeline2(data, count_letter=True):
    start_time = time.time()
    count = 0
    for bad_cut in data:
        count += 1
        bad_cuts = np.asarray([bad_cut])
        if count_letter:
            print ("Predictions for the supposed letter number " + str(count))
        predictions = segmentator_model.predict([bad_cuts] * number_of_models)
        if (predictions[0][1] > predictions[0][0]):
            predictions = ocr_model.predict([bad_cuts] * number_of_models)
            ind = (-predictions[0]).argsort()[:3]
            for i in ind:
                print ("Good cut corresponding to letter '" + ALPHABET_ALL[i] + \
                   "' with a confidence of " + str(predictions[0][i] * 100) + "%")
        else:
            print ("Bad cut with a confidence of " + str(predictions[0][0] * 100) + "%")
        print ("---")
    elapsed_time = time.time() - start_time
    print("Elapsed time:", elapsed_time)

def predict_pipeline3(data, count_letter=True):
    start_time = time.time()
    count = 0
    for bad_cut in data:
        flag = False
        count += 1
        bad_cuts = np.asarray([bad_cut])
        if count_letter:
...


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Loading test sets
Can you write Python code for it?

# letter list
ALPHABET_ALL = dataset.ALPHABET_ALL

(_, _, X_test_22, y_test_22, _) = dataset.generate_all_chars_with_class(verbose=0, plot=False)
(input_shape, X_test_22, Y_test_22) = kiu.adjust_input_output(X_test_22, y_test_22, 22)
print ("Loaded test set for all the characters")

(_, _, X_test_seg, y_test_seg) = dataset.generate_dataset_for_segmentator(verbose=0, plot=False)
(_, X_test_seg, Y_test_seg) = kiu.adjust_input_output(X_test_seg, y_test_seg, 2)
print ("Loaded test set for good and bad segments")

X_test_char = {}
y_test_char = {}
Y_test_char = {}
for char in ALPHABET_ALL:
    (_, _, X_test_char[char], y_test_char[char]) = dataset.generate_positive_and_negative_labeled(char, verbose=0)
    (_, X_test_char[char], Y_test_char[char]) = kiu.adjust_input_output(X_test_char[char], y_test_char[char], 2)
    print ("Loaded test set for char '" + char + "'")


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Pipeline 1 (22 networks as segmentator and classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word:  ----s
Good cuts
Can you write Python code for it?

asseras_good_cuts = iu.open_many_samples( \
                    ["not_code/words/good_cuts/asseras/a1.png",
                     "not_code/words/good_cuts/asseras/f1.png",
                     "not_code/words/good_cuts/asseras/f2.png",
                     "not_code/words/good_cuts/asseras/e.png",
                     "not_code/words/good_cuts/asseras/r.png",
                     "not_code/words/good_cuts/asseras/a2.png",
                     "not_code/words/good_cuts/asseras/s.png"])
(asseras_good_cuts, _) = kiu.adjust_input(np.asarray(asseras_good_cuts))

u.plot_some_images(asseras_good_cuts, (img_cols, img_rows), grid_x=7, grid_y=1)


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word:  ---ef--i
Good cuts
Can you write Python code for it?

beneficiu_good_cuts = iu.open_many_samples( \
                    ["not_code/words/good_cuts/beneficiu/b.png",
                     "not_code/words/good_cuts/beneficiu/e1.png",
                     "not_code/words/good_cuts/beneficiu/n.png",
                     "not_code/words/good_cuts/beneficiu/e2.png",
                     "not_code/words/good_cuts/beneficiu/f.png",
                     "not_code/words/good_cuts/beneficiu/i1.png",
                     "not_code/words/good_cuts/beneficiu/c.png",
                     "not_code/words/good_cuts/beneficiu/i2.png",
                     "not_code/words/good_cuts/beneficiu/u.png"])
(beneficiu_good_cuts, _) = kiu.adjust_input(np.asarray(beneficiu_good_cuts))

u.plot_some_images(beneficiu_good_cuts, (img_cols, img_rows), grid_x=3, grid_y=3)


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word:  unicu
Experiment 3 ("beneficiu")
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: ----s
Pipeline 3 (22 networks as segmentator + classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: -ls-s
Pipeline 2 (segmentator + classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: -uu--
Pipeline 3 (22 networks as segmentator + classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: asseras
Pipeline 2 (segmentator + classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: asseras
Pipeline 4 (segmentator + 22 networks as classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: benes-ciu or benef-ciu with a lower chance
Pipeline 3 (22 networks as segmentator + classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: beuessciu
Pipeline 2 (segmentator + classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: biiefoii
Pipeline 4 (segmentator + 22 networks as classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: iuuci
Pipeline 2 (segmentator + classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: siiescii
Pipeline 2 (segmentator + classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: unicu
Pipeline 2 (segmentator + classifier)
Can you write Python code for it?


I am working on the file "Notebooks/11_ICR-Test_letter-recognition.ipynb".
The first task is:
Possible word: unicu
Pipeline 3 (22 networks as segmentator + classifier)
Can you write Python code for it?


I am working on the file "Notebooks/16_Step_12.ipynb".
The first task is:
For the meat of our computation, we're going to reach back to a trick we used in Step 9 for Laplace's Equation.  We're interested in what our grid will look like once we've reached a near-steady state.  We can either specify a number of timesteps nt and increment it until we're satisfied with the results, or we can tell our code to run until the difference between two consecutive iterations is very small.
We also have to manage 8 separate boundary conditions for each iteration.  The code below writes each of them out explicitly.  If you're interested in a challenge, you can try to write a function which can handle some or all of these boundary conditions.  If you're interested in tackling that, you should probably read up on Python dictionaries.
Can you write Python code for it?


I am working on the file "Notebooks/16_Step_12.ipynb".
The first task is:
If you want to see how the number of iterations increases as our udiff condition gets smaller and smaller, try defining a function to perform the while loop written above that takes an input udiff and outputs the number of iterations that the function runs.
For now, let's look at our results.  We've used the quiver function to look at the cavity flow results and it works well for channel flow, too.
Can you write Python code for it?


I am working on the file "Notebooks/16_Step_12.ipynb".
The first task is:
In step 11, we isolated a portion of our transposed equation to make it easier to parse and we're going to do the same thing here.  One thing to note is that we have periodic boundary conditions throughout this grid, so we need to explicitly calculate the values at the leading and trailing edge of our u vector.
Can you write Python code for it?


I am working on the file "Notebooks/16_Step_12.ipynb".
The first task is:
Now we have our familiar list of variables and initial conditions to declare before we start.
Can you write Python code for it?

##variable declarations
nx = 41
ny = 41
nt = 10
nit = 50
c = 1
dx = 2 / (nx - 1)
dy = 2 / (ny - 1)
x = numpy.linspace(0, 2, nx)
y = numpy.linspace(0, 2, ny)
X, Y = numpy.meshgrid(x, y)

##physical variables
rho = 1
nu = .1
F = 1
dt = .01

#initial conditions
u = numpy.zeros((ny, nx))
un = numpy.zeros((ny, nx))

v = numpy.zeros((ny, nx))
vn = numpy.zeros((ny, nx))

p = numpy.ones((ny, nx))
pn = numpy.ones((ny, nx))

b = numpy.zeros((ny, nx))


I am working on the file "Notebooks/16_Step_12.ipynb".
The first task is:
The structures in the quiver command that look like [::3, ::3] are useful when dealing with large amounts of data that you want to visualize.  The one used above tells matplotlib to only plot every 3rd data point.  If we leave it out, you can see that the results can appear a little crowded.
Can you write Python code for it?

fig = pyplot.figure(figsize = (11,7), dpi=100)
pyplot.quiver(X, Y, u, v);


I am working on the file "Notebooks/16_Step_12.ipynb".
The first task is:
We'll also define a Pressure Poisson iterative function, again like we did in Step 11.  Once more, note that we have to include the periodic boundary conditions at the leading and trailing edge.  We also have to specify the boundary conditions at the top and bottom of our grid.
Can you write Python code for it?


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
Alternatively, we could loop over all files and split on the / character.  This will create a new list where each element is whatever characters are separated by the splitting character.  We can then take the last element of each list.
Can you write Python code for it?

file_list = []
for f in data_file_list:
    file_list.append(f.split('/')[-1])

print(file_list)


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
Conditional Logic (if...)
Unlike most other languages, Python uses tab formatting rather than closing conditional statements (e.g., end).

Syntax:

python
if condition:
    do something

Implicit conversion of the value to bool() happens if condition is of a different type than bool, thus all of the following should work:

python
if condition:
    do_something
elif condition:
    do_alternative1
else:
    do_otherwise # often reserved to report an error
                 # after a long list of options
Can you write Python code for it?

n = 1

if n:
    print("n is non-0")

if n is None:
    print("n is None")

if n is not None:
    print("n is not None")


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
Dictionaries

In Python, a dictionary (or dict) is mapping between a set of
indices (keys) and a set of values

The items in a dictionary are key-value pairs

Keys can be any Python data type

Dictionaries are unordered

Here is a more indepth tutorial on dictionaries
Can you write Python code for it?

# Dictionaries
eng2sp = {}
eng2sp['one'] = 'uno'
print(eng2sp)

eng2sp = {'one': 'uno', 'two': 'dos', 'three': 'tres'}
print(eng2sp)

print(eng2sp.keys())
print(eng2sp.values())


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
Functions
A function is a named sequence of statements that performs a computation.  You define the function by giving it a name, specify a sequence of statements, and optionally values to return.  Later, you can “call” the function by name.
python
def make_upper_case(text):
    return (text.upper())
* The expression in the parenthesis is the argument.
* It is common to say that a function “takes” an argument and “returns” a result.
* The result is called the return value.
The first line of the function definition is called the header; the rest is called the body.
The header has to end with a colon and the body has to be indented.
It is a common practice to use 4 spaces for indentation, and to avoid mixing with tabs.
Function body in Python ends whenever statement begins at the original level of indentation.  There is no end or fed or any other identify to signal the end of function.  Indentation is part of the the language syntax in Python, making it more readable and less cluttered.
Can you write Python code for it?

def make_upper_case(text):
    return (text.upper())

string = "Python is going to make conducting research easier"

print(make_upper_case(string))


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
It is also sometimes even cleaner to do this as a list comprehension
Can you write Python code for it?

[os.path.basename(x) for x in data_file_list]


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
It looks like we are currently in the notebooks folder of the github repository.  Let's use glob, a pattern matching function, to list all of the csv files in the Data folder.
Can you write Python code for it?

data_file_list = glob(os.path.join('..','Data','*csv'))
print(data_file_list)


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
Math Operators

+, -, *, and /
Exponentiation **

Modulo %

Note that division with integers in Python 2.7 automatically rounds, which may not be intended.  It is recommended to import the division module from python3 from __future__ import division
Can you write Python code for it?

# Addition
a = 2 + 7
print(a)

# Subtraction
b = a - 5
print(b)

# Multiplication
print(b*2)

# Exponentiation
print(b**2)

# Modulo
print(4%9)

# Division
print(4/9)


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
Package Management
Package managment in Python has been dramatically improving.  Anaconda has it's own package manager called 'conda'.  Use this if you would like to install a new module as it is optimized to work with anaconda.
!conda install *package*
However, sometimes conda doesn't have a particular package.  In this case use the default python package manager called 'pip'.
These commands can be run in your unix terminal or you can send them to the shell from a Jupyter notebook by starting the line with !
It is easy to get help on how to use the package managers
!pip help install
Can you write Python code for it?

!pip help install

!pip list --outdated

!pip install setuptools --upgrade


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
Python Containers
There are 4 main types of builtin containers for storing data in Python:
* list
* tuple
* dict
* set
Lists
In Python, a list is a mutable sequence of values.  Mutable means that we can change separate entries within a list. For a more in depth tutorial on lists look here

Each value in the list is an element or item
Elements can be any Python data type

Lists can mix data types

Lists are initialized with [] or list()
python
l = [1,2,3]

Elements within a list are indexed (starting with 0)
python
l[0]

Elements can be nested lists
python
nested = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

Lists can be sliced.
python
l[start:stop:stride]

Like all python containers, lists have many useful methods that can be applied
python
a.insert(index,new element)
a.append(element to add at end)
len(a)

List comprehension is a Very powerful technique allowing for efficient construction of new lists.
python
[a for a in l]
Can you write Python code for it?

# Indexing and Slicing
a = ['lists','are','arrays']
print(a[0])
print(a[1:3])

# List methods
a.insert(2,'python')
a.append('.')
print(a)
print(len(a))

# List Comprehension
print([x.upper() for x in a])


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
String Operators

Some of the arithmetic operators also have meaning for strings. E.g. for string concatenation use + sign
String repetition: Use * sign with a number of repetitions
Can you write Python code for it?

# Combine string
a = 'Hello'
b = 'World'
print(a + b)

# Repeat String
print(a*5)


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
Tuples
In Python, a tuple is an immutable sequence of values, meaning they can't be changed

Each value in the tuple is an element or item

Elements can be any Python data type

Tuples can mix data types

Elements can be nested tuples

Essentially tuples are immutable lists

Here is a nice tutorial on tuples
Can you write Python code for it?

numbers = (1, 2, 3, 4)
print(numbers)

t2 = 1, 2
print(t2)


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
Variables

Built-in
Numeric types:
int, float, long, complex

string
boolean
True / False

NoneType

User defined

Use the type() function to find the type for a value or variable

Data can be converted using cast commands
Can you write Python code for it?

# Integer
a = 1
print(type(a))

# Float
b = 1.0
print(type(b))

# String
c = 'hello'
print(type(c))

# Boolean
d = True
print(type(d))

# None
e = None
print(type(e))

# Cast integer to string
print(type(str(a)))


I am working on the file "Notebooks/1_Introduction_to_Programming.ipynb".
The first task is:
sets
In Python, a set is an efficient storage for "membership" checking

set is like a dict but only with keys and without values

a set can also perform set operations (e.g., union intersection)

Here is more info on sets
Can you write Python code for it?


I am working on the file "Notebooks/1_lengthDistributions.ipynb".
The first task is:
Even fewer of the reference introns overlapping at short distances - maybe the known utrons which overlap at very short distances are actually deletions / not real
Can you write Python code for it?

# Plot Vals
pylab.hist(a.tolist(), range=(0,400), bins=50, color="red", normed=True, label="Reference Introns", edgecolor = "black")
pylab.hist(allLengths, range=(0,400), bins=50, color="blue", normed=True, alpha=0.7, label="All UTRons", edgecolor = "black")

# Figure Options
pylab.legend(fontsize="small")
pylab.xlabel("Length"); pylab.ylabel("Normalised Distribution")
pylab.savefig("./images/1_AllVsReference_400", dpi=300)

# Plot Vals
pylab.hist(allLengths, range=(250,1000), bins=30, color="red", normed=True, label="Annotated UTRons",edgecolor = "black")
pylab.hist(novelLengths, range=(250,1000), bins=30, color="blue", normed=True, alpha=0.7, label="Novel UTRons",edgecolor = "black")

# Figure Options
pylab.legend(fontsize="small")
pylab.xlabel("Length"); pylab.ylabel("Normalised Distribution")
pylab.savefig("./images/1_NovelVsKnown_250-1000", dpi=300)


I am working on the file "Notebooks/1_lengthDistributions.ipynb".
The first task is:
When looking at distr. of 0-400bp this confirms that lots of the utrons are short wheras only a few of the annotated UTRons are at short lengths
Can you write Python code for it?

# Plot vals
pylab.hist(a.tolist(), range=(0,400), bins=50, color="red", normed=True, label="Reference Introns",edgecolor = "black")
pylab.hist(novelLengths, range=(0,400), bins=50, color="blue", normed=True, alpha=0.7, label="Novel UTRons",edgecolor = "black")

# Figure Options
pylab.legend(fontsize="small")
pylab.xlabel("Length"); pylab.ylabel("Normalised Distribution")
pylab.savefig("./images/1_NovelVsReference_400", dpi=300)


I am working on the file "Notebooks/2-Playing with Images.ipynb".
The first task is:
If you open the corresponding image in Fiji you will see that its displayed in grayscale, while here it looks blue-green. This is because the imshow() function applies something called as a lookup table (LUT) to help our eyes see the high and low intensities better. If you would like to still view it as a grey image we can change the display parameters as below:
Can you write Python code for it?

import matplotlib.cm as cm
imshow(in_img, cmap=cm.gray)


I am working on the file "Notebooks/2-Playing with Images.ipynb".
The first task is:
Note: Regionprops returns many properties a full list of the properties is available in the help file. The help for any imported function can be accessed by running the help(regionprops) command.
Can you write Python code for it?


I am working on the file "Notebooks/2-Playing with Images.ipynb".
The first task is:
Now we have a way to determine the order in which objects are measured. The function to measure objects is called regionprops(). This function takes in the labelled image and the intensity image as an input and returns an object which has measurements of each of the objects.
Can you write Python code for it?

from skimage.measure import regionprops

r_props = regionprops(label_im, in_img)

#print length of number r_props to determine number of objects measured.
len(r_props)


I am working on the file "Notebooks/2-Playing with Images.ipynb".
The first task is:
Reading Images
In the last cell we printed the name of the first image file in the Type1 directory. We will use this as our test file for now.
To read images anaconda comes with the sci-kit image package, which has many useful functions for image analysis. Among them are the functions imread() and imshow() these readn and display the image respectively. As with os and glob the scikit image package has to be imported, but in this case we won't be using many functions from the package so we will only import the imread() and imshow() functions.
you will notice a strange looking line: %matplotlib inline this is actually meant for the jupyter notebook. This tells the notebook to display the images within the page.
Can you write Python code for it?

from skimage.io import imread, imshow
%matplotlib inline
in_img = imread(file_paths[0][0])
imshow(in_img)


I am working on the file "Notebooks/2-Playing with Images.ipynb".
The first task is:
Unlike the other data types which we have seen so far regionprops gives us a list of named properties these can be accessed as below:
Can you write Python code for it?

# Area for the first object
print(r_props[0].area)
# Area for the first object
print(r_props[1].area)
# Area for the first object
print(r_props[2].area)


I am working on the file "Notebooks/2-Playing with Images.ipynb".
The first task is:
We can also use list comprehension to collect all the measured areas into a single list.
Can you write Python code for it?

all_areas = [rp.area for rp in r_props]
print(all_areas)


I am working on the file "Notebooks/2-Playing with Images.ipynb".
The first task is:
We get a boolean array which is full of true and false values corresponding to our question: is this number bigger than 1
Boolean arrays have some interesting properties, one of which is that you can multiply them with regular arrays. The True values are treated as 1 and the False as 0. However, we will not go into that right now.
Right now we'd like to know of a good method to identify our nuclei.
Otsu: the man on the threshold.
In the last section when we asked the computer which values are greater than 1 what we did was set the threshold as '1'. But this isn't always straighforward especially with images where the intensities can vary a lot. This means that if you painfully find the threshold for one image there is no gurantee that it will work for the next image. What we need to do is to use the characteristics of the image to determine what the appropriate threshold is.
Otsu's method is a very effective way of doing this. I won't go into the details of the method but I highly reccomend reading about it: https://en.wikipedia.org/wiki/Otsu%27s_method
The nice people behind the Scikit Image package have already made a function to determine the otsu threshold as shown below. This can alse be applied to our image.
Can you write Python code for it?

from skimage.filters import threshold_otsu
my_array_thresh = threshold_otsu(myarray)
print(my_array_thresh)

print(myarray>my_array_thresh)
imshow(myarray>my_array_thresh)

img_thresh = threshold_otsu(in_img)
print(img_thresh)
bw_img = in_img>img_thresh
imshow(bw_img)


I am working on the file "Notebooks/2-Playing with Images.ipynb".
The first task is:
With the image we would like to find all the pixels that are bright. In the case of our small array, we can decide that everything bigger than 1 is a bright pixel. Using this as our criteria, we can as the computer to identify the pixels for us.
How you ask? Remember the booleans from the last lesson?
Can you write Python code for it?


I am working on the file "Notebooks/2-Playing with Images.ipynb".
The first task is:
You might notice that the object passed to the np.array() function is a list. The np.array() function takes the list and converts it to an array. You know it's an array because it says 'array' when its printed. This is the same thing that was printed when we called our image variable. So, we can safely say that our image is also an array. Anything that we do with this small array here can also be done to our bigger image array. In fact we can even treat our little array as an image and see how it looks:
Can you write Python code for it?


I am working on the file "Notebooks/2.1_MatplotlibTemplates.ipynb".
The first task is:
Bar graphs with standard error bars for 2 group
Can you write Python code for it?

# Make some fake data
d = {'race': np.random.permutation(np.hstack([np.ones(10), np.zeros(10)])),
     'gender': np.hstack([np.ones(10), np.zeros(10)]),
     'scores': np.hstack([np.random.rand(10), np.random.rand(10)+1])}
df = pd.DataFrame(d)

# Change this part and replace with the variables you want to plot and the grouping variable column name.
val =['scores']
group1 = ['gender']
group2 = ['race']

# Get means and sems for Gender group
means1 = df[val+group1].groupby(group1).mean().squeeze()
sems1 = df[val+group1].groupby(group1).sem().squeeze()
# Get means and sems for Race group
means2 = df[val+group2].groupby(group2).mean().squeeze()
sems2 = df[val+group2].groupby(group2).sem().squeeze()

fig,ax = plt.subplots(figsize=(10,5)) # Change figure size in (width,height)
ind = np.array([0.,1.]) # location of bars
width = .4 # Width of bars

# plot score by gender
rects1 = ax.bar(ind - width,means1,width,color='lightcoral',yerr=sems1,ecolor='k') # (bar x-location, bar heights, width=bar width, color=bar color, yerr=standard error)
# plot score by race
rects2 = ax.bar(ind,means2,width,color='lightblue',yerr=sems2,ecolor='k')

# configure axes properties to make pretty
ax.set_ylabel('scores')
ax.set_xlabel('gender')
ax.set_title('Scores by gender and race')
ax.set_xticks(ind)
ax.set_xticklabels(['Male','Female'])
ax.set_xlim([ind[0]-width*1.25,ind[-1]+width*1.25])
ax.set_ylim([0,1.8])

ax.legend(['Race0','Race1'])

autolabel(rects1)
autolabel(rects2)


I am working on the file "Notebooks/2.1_MatplotlibTemplates.ipynb".
The first task is:
Drawing trend line on a scatterplot
Can you write Python code for it?

import statsmodels.formula.api as smf
import statsmodels.api as sm

d = {'race': np.random.permutation(np.hstack([np.ones(20), np.zeros(20)])),
     'gender': np.hstack([np.ones(20), np.zeros(20)]),
     'scores': np.round(10*np.hstack([np.random.rand(20), np.random.rand(20)+1]))}
df = pd.DataFrame(d)
lm = smf.ols(formula = "scores ~ gender",data=df).fit()
print(lm.summary())

# Save the slope for gender to b1 and intercept to b0
b1 = lm.params[1] # This is slope
b0 = lm.params[0] # This is intercept

# Set x,y values for each group
gender0 = 0 # value of first group
y0 = df[['scores']].loc[df['gender']==gender0].values.squeeze()
y0 = y0+(np.random.rand(len(y0))-.5)*.1 #Change after + sign to control dispersion
x0 = np.ones(len(y0))*gender0 + (np.random.rand(len(y0))-.5)*.1 #Change after + sign to control dispersion

gender1 = 1 # value of second group
y1 = df[['scores']].loc[df['gender']==gender1].values.squeeze()
y1 = y1+(np.random.rand(len(y1))-.5)*.1
x1 = np.ones(len(y1))*gender1 + (np.random.rand(len(y1))-.5)*.1

fig,ax = plt.subplots(figsize=(5,5))
ax.scatter(x0,y0,color='lightcoral')
ax.scatter(x1,y1,color='lightcoral')

# Part that adds the line
spacing = 10
minx = df[['gender']].min().squeeze()
maxx = df[['gender']].max().squeeze()
lx = np.linspace(minx,maxx,spacing) # make x coordinates
ly = b0+lx*b1 # Estimate the y values using betas
ax.plot(lx,ly,'-k')

ax.set_ylabel('scores')
ax.set_xlabel('gender')
ax.set_title('Values are now dispersed')
ax.set_xticks([0,1])
ax.set_xticklabels(['Male','Female'])
ax.set_xlim([-.5,1.5])
ax.grid()
plt.show()


I am working on the file "Notebooks/2.1_MatplotlibTemplates.ipynb".
The first task is:
Here is the fix.
Can you write Python code for it?

# Set x,y values for each group
gender0 = 0 # value of first group
y0 = df[['scores']].loc[df['gender']==gender0].values.squeeze() # Grabs y values for Gender =0
y0 = y0+(np.random.rand(len(y0))-.5)*.1 #Change after + sign to control dispersion
x0 = np.ones(len(y0))*gender0 +(np.random.rand(len(y0))-.5)*.1 #Change after + sign to control dispersion

gender1 = 1 # value of second group
y1 = df[['scores']].loc[df['gender']==gender1].values.squeeze()
y1 = y1+(np.random.rand(len(y1))-.5)*.1
x1 = np.ones(len(y1))*gender1 + (np.random.rand(len(y1))-.5)*.1

fig,ax = plt.subplots(figsize=(5,5))
ax.scatter(x0,y0,color='lightcoral')
ax.scatter(x1,y1,color='lightcoral')
ax.set_ylabel('scores')
ax.set_xlabel('gender')
ax.set_title('Values are now dispersed')
ax.set_xticks([0,1])
ax.set_xticklabels(['Male','Female'])
ax.set_xlim([-.5,1.5])
ax.grid() # puts grid on
plt.show()


I am working on the file "Notebooks/2.1_MatplotlibTemplates.ipynb".
The first task is:
Scatterplots of 1 group with jittered location
If you try to plot something like a scaled data, you won't be able to see how clustered they are because they would just plot on top of each other. One way to avoid this is to jitter the x,y locations around the actual value.
Can you write Python code for it?

# Make some fake data
d = {'race': np.random.permutation(np.hstack([np.ones(20), np.zeros(20)])),
     'gender': np.hstack([np.ones(20), np.zeros(20)]),
     'scores': np.round(10*np.hstack([np.random.rand(20), np.random.rand(20)+1]))}
df = pd.DataFrame(d)
ax = df.plot(kind='scatter',x='gender',y='scores')
ax.set_title('Values are stacked')
plt.show()


I am working on the file "Notebooks/2.1_MatplotlibTemplates.ipynb".
The first task is:
Simple Plots in Python
In this tutorial we'll show you some basic templates of scientific plots using Python matplotlib.
Bar graphs with standard error bars for 1 group
Can you write Python code for it?

# based on http://matplotlib.org/examples/api/barchart_demo.html

# Make some fake data
d = {'gender': np.hstack([np.ones(10), np.zeros(10)]), 'scores': np.hstack([np.random.rand(10), np.random.rand(10)+1])}
df = pd.DataFrame(d)

# Change this part and replace with the variables you want to plot and the grouping variable column name.
vals = ['scores'] # This is the column name of the variable to plot on Y axis
group = ['gender'] # This is the grouping variable for the X axis

# Get means for each group
means = df[vals+group].groupby(group).mean().squeeze()
# Get standard error of means for each group
sems = df[vals+group].groupby(group).sem().squeeze()

fig,ax = plt.subplots(figsize=(10,5)) # Change figure size in (width,height)
ind = np.arange(np.size(np.unique(df[group]),0)) # location of bars
width = .5 # Width of bars
# (bar x-location, bar heights, width=bar width, color=bar color, yerr=standard error,ecolor=errorbarcolor)
rects1 = ax.bar(ind - width/2,means,width=.5,color='lightsalmon',yerr=sems,ecolor='blue')
# Look up different colors here: http://stackoverflow.com/questions/22408237/named-colors-in-matplotlib

# configure axes properties to make pretty
ax.set_ylabel('scores')
ax.set_xlabel('gender')
ax.set_title('Scores by gender')
ax.set_xticks(ind)
ax.set_xticklabels(['Male','Female'])
ax.set_xlim([-.5,1.5])
ax.set_ylim([0,2])

# This part calls the function autolabel() defined above, and labels the bars with values
autolabel(rects1)

plt.show()


I am working on the file "Notebooks/2015_04_16_AL_Analyse_cross_conta_data_Pierre.ipynb".
The first task is:
Create lists to store results for different thresholds from 0 to 25%
Can you write Python code for it?

res_s1_sup_s2 = [0 for i in range (26)]
res_s2_sup_s1 = [0 for i in range (26)]
res_s1_sup_other = [0 for i in range (26)]
res_s2_sup_other = [0 for i in range (26)]


I am working on the file "Notebooks/2015_04_16_AL_Analyse_cross_conta_data_Pierre.ipynb".
The first task is:
Import packages
Can you write Python code for it?

%pylab inline
import csv
import matplotlib.pyplot as plt


I am working on the file "Notebooks/2015_04_16_AL_Analyse_cross_conta_data_Pierre.ipynb".
The first task is:
parse file and populate the list
Can you write Python code for it?

with open('./jeter.tsv', 'r') as csvfile:
    reader = csv.reader(csvfile, delimiter='\t')
    # remove header
    next(reader)
    # iterate over rows
    for R in reader:
        s1_sup_s2 = float(R[6])*100/(float(R[5])+float(R[6])+float(R[7]))
        s2_sup_s1 = float(R[9])*100/(float(R[8])+float(R[9])+float(R[10]))
        s1_sup_other = float(R[7])*100/(float(R[5])+float(R[6])+float(R[7]))
        s2_sup_other = float(R[10])*100/(float(R[8])+float(R[9])+float(R[10]))
        for seuil in range (26):
            if s1_sup_s2 >= seuil:
                res_s1_sup_s2[seuil]+=1
            if s2_sup_s1 >= seuil:
                res_s2_sup_s1[seuil]+=1
            if s1_sup_other >= seuil:
                res_s1_sup_other[seuil]+=1
            if s2_sup_other >= seuil:
                res_s2_sup_other[seuil]+=1

print (res_s1_sup_s2)
print (res_s2_sup_s1)
print (res_s1_sup_other)
print (res_s2_sup_other)

plt.figure(figsize=(20, 10))
plt.title("percentage of samples with a number of read with incorect an genotype corresponding to another sample from the same lane or a randon error")
plt.xlabel("Percentage of sample")
plt.ylabel("Number of read with incorrect genotype")
plt.ylim(1,77869)

line1 = plt.semilogy(res_s1_sup_s2, 'b', label = 'reads_sample1_supporting_sample2')
line2 = plt.semilogy(res_s2_sup_s1, 'g', label = 'reads_sample2_supporting_sample1')
line3 = plt.semilogy(res_s1_sup_other, 'r', label = 'reads_sample1_supporting_others')
line4 = plt.semilogy(res_s2_sup_other, 'm', label = 'reads_sample2_supporting_others')

plt.legend(loc='best')


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
1 nt wide peaks = No problem with this dataset. Since there are no gene loci, I just filed the field with a dash to indicate that it is empty

m6A_Dominissini_hg38
Can you write Python code for it?

infile="./PTM_Original_Datasets/m6A_Dominissini_hg38.bed"
PMID = "22575960"
cell = "HepG2"
modification = "m6A"
method = "M6A_seq"
author = "Dominissini"
outfile = "./PTM_Clean_Datasets/{}_{}_{}_hg38_cleaned.bed".format(author, modification, cell, method)

file_summary(infile)

distrib_peak_len(infile, normed=False, bins=500)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
Add the gene name and ensembl gene ID to the bed name field
Test with one file
Can you write Python code for it?

annotation_file = "../../Reference_Annotation/gencode_v24.gff3"
peak_file = "./PTM_Clean_Datasets/DARNED_human_hg38_inosine_cleaned.bed"
outdir = "./"

output_file = "./DARNED_human_hg38_inosine_cleaned_gencode_v24.bed"
genecount_file = "./DARNED_human_hg38_inosine_cleaned_gencode_v24_uniq-gene.csv"
site_file = "./DARNED_human_hg38_inosine_cleaned_gencode_v24_uniq-sites.csv"

intersect_extract_genecodeID (annotation_file, peak_file, outdir)
file_summary(output_file, separator=["\t","|"])

head (genecount_file)
head (site_file)

remove(output_file)
remove(genecount_file)
remove(site_file)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
After combining information, out of the 1343463 sites in the RADAR secondary file and 2576460 in the RADAR primary file, 1342423 consistant sites were found in both files, ie half of the database site were filtered out because they where not in the 2 database files.
Convert RADAR coordinates from hg19 to hg39 with Crossmap
Coordinate conversion using CrossMap in BASH
Can you write Python code for it?

# Conversion to hg38 with Crossmap/liftover
lifover_chainfile = "../LiftOver_chain_files/hg19ToHg38.over.chain.gz"
input_bed = "./PTM_Original_Datasets/RADAR_Human_hg19_inosine_cleaned.bed"
temp_bed = "./PTM_Original_Datasets/RADAR_Human_hg38_inosine_temp.bed"

cmd = "CrossMap.py bed {} {} {}".format(lifover_chainfile, input_bed, temp_bed)
bash(cmd)

# Rewriting and updating of the header removed by Crossmap
final_bed = "./PTM_Original_Datasets/RADAR_Human_hg38_inosine_cleaned.bed"

header = "# RADAR database Human v2 all sites hg38 coordinates\n"
header+= "# Data cleaned, standardized, converted to BED6 format and updated to hg38 coordinates\n"
header+= "# Adrien Leger (aleg@ebi.ac.uk) - {}\n".format(str (datetime.datetime.today()))
header+= "# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

with open (temp_bed, "r") as infile, open (final_bed, "w") as outfile:
    outfile.write (header)
    for line in infile:
        outfile.write (line)

file_summary(final_bed)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
Between 1 and 16% (excluding the weird carlile dataset) of the peaks are found in lncRNA annotated in gencode v24.
Iterate through the datasets to find the number of uniq genes found
Can you write Python code for it?

# Annotation vs gencodev23 All genes
annotation_file = "/home/aleg/Data/Reference_Annotation/gencode_v24.gff3"

for peak_file in sorted(glob("./PTM_Clean_Datasets/*.bed")):
    outdir = "./PTM_Annotated_Datasets"
    intersect_extract_genecodeID (annotation_file, peak_file, outdir)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
Contains a lot of fields, some of which I don't even have an idea of what the contains. The dataset was not filtered and contains not only A>G A>I transitions. There is a total of 22686 sites but only 21111 are A>G transitions. I filtered out all the other modifications and retained only the A>G transition.
I am not sure that I should use this dataset, especially since it is already included in the RADAR database.

editing_Sakurai_hg38
Can you write Python code for it?

infile = "./PTM_Original_Datasets/editing_Sakurai_hg38.bed"
PMID = "24407955"
cell = "Brain"
modification = "A>I"
method = "ICE_seq"
author = "Sakurai"
outfile = "./PTM_Clean_Datasets/{}_{}_{}_hg38_cleaned.bed".format(author, modification, cell, method)

file_summary(infile)

# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

init_template=[0,"\t",1,"\t",2,"\t",3,"|",4,"|",5,"|",6,"\t",7,"\t",8]
final_template=[0,"\t",1,"\t",2,"\t",modification,"|",cell,"|",method,"|",PMID,"|",4,"\t",7,"\t",8]

reformat_table(
    input_file=infile,
    output_file=outfile,
    init_template=init_template,
    final_template=final_template,
    keep_original_header = False,
    header = generate_header(PMID, cell, modification, method),
    replace_internal_space='_',
    replace_null_val="-")

file_summary(outfile)

distrib_peak_len(outfile)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
DARNED is a little messy and hard to convert since the position with the same PMID/OR sample types where fused in the same Site. It makes it difficult to parse. I think it would be better if I duplicate the site with the several PMID and sample type. I just need to verify if the number of fields in PMID and cell type is similar and if they correspond to each other, ie first in cell type = first in PMID
Can you write Python code for it?

# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

header = "# DARNED database Human all sites hg19 coordinates\n"
header+= "# Data cleaned, filtered for Inosine editing, standardized and converted to BED6 format\n"
header+= "# Adrien Leger (aleg@ebi.ac.uk) - {}\n".format(str (datetime.datetime.today()))
header+= "# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

reformat_table(
    input_file = "./PTM_Original_Datasets/DARNED_human_hg19_all_sites.txt",
    output_file = "./PTM_Original_Datasets/DARNED_human_hg19_inosine.bed",
    init_template = [0,"\t",1,"\t",2,"\t",3,"\t",4,"\t",5,"\t",6,"\t",7,"\t",8,"\t",9],
    final_template = ["chr",0,"\t",1,"\t",1,"\t",3,">",4,"|",8,"|-|",9,"|",5,"\t0\t",2],
    keep_original_header = False,
    header = header,
    replace_internal_space = '_',
    replace_null_val = "-",
    filter_dict = {0:["-"],1:["-"],2:["?"],3:["T","G","C"],4:["C","T","A","U"],8:["-"],9:["-"]},
    subst_dict = {4:{"G":"I"}}
    )

file_summary("./PTM_Original_Datasets/DARNED_human_hg19_inosine.bed")


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
Dataset OK but only contains only 8 peaks in lncRNA. 1 of the peaks is really wide = 40000 pb The dataset contains only the 8 peaks identify in the lncRNA... The coordinates correspond to the gen coordinates rather than the peaks themselves.. Use the dataset?? Is it really worthy to remap everything for such a low number of eventual peaks?
pseudoU_Li_hg38
Can you write Python code for it?

infile="./PTM_Original_Datasets/pseudoU_Li_hg38.bed"
PMID = "26075521"
cell = "HEK293"
modification = "Y"
method = "CeU_Seq"
author = "Li"
outfile = "./PTM_Clean_Datasets/{}_{}_{}_hg38_cleaned.bed".format(author, modification, cell, method)

file_summary(infile)

# chrom	chromstart	chromend	modif|cell_type|method|PMID|loci	score	strand\n"

init_template=[0,"\t",1,"\t",2,"\t",3,"|",4,"|",5,"|",6,"\t",7,"\t",8]
final_template=[0,"\t",1,"\t",2,"\t",modification,"|",cell,"|",method,"|",PMID,"|",4,"\t",7,"\t",8]

reformat_table(
    input_file=infile,
    output_file=outfile,
    init_template=init_template,
    final_template=final_template,
    keep_original_header = False,
    header = generate_header(PMID, cell, modification, method),
    replace_internal_space='_',
    replace_null_val="-")

file_summary(outfile)

distrib_peak_len(outfile)


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
Gene annotation of the PTM datasets
 The original annotations might not be optimal, and probably not made from an uniq reference annotation file. I will reanotate all the datasets will the last version of gencodegencode.v24.long_noncoding_RNAs.gff3. I split the file in 3 to retain only genes, transcript and exons. I also got the general gencode file containing all the annotated genes in the primary assembly

gencodegencode.v24.long_noncoding_RNAs = Contains the comprehensive gene annotation of lncRNA genes on the reference chromosomes
gencode.v24.annotation = Contains the comprehensive gene annotation on the primary assembly (chromosomes and scaffolds) sequence regions
Can you write Python code for it?

# New dir to create annotated files
mkdir("PTM_Annotated_Datasets")
mkdir("Test")


I am working on the file "Notebooks/2016_05_23_Epi2_pilot_datasets_analysis_PTM.ipynb".
The first task is:
I found a python wrapper package for bedtools to manipulate bed files. I will use it to intersect my bed files containing the positions of the PTM (or peaks) and the gff3 annotation files. This will allow me to get gene names for each positions
Can you write Python code for it?

help(reformat_table)

head("../../Reference_Annotation/gencode_v24.gff3.gz")

import pybedtools
annotation_file = "../../Reference_Annotation/gencode_v24.gff3"
peak_file = "./PTM_Clean_Datasets/Dominissini_m6A_HepG2_hg38_cleaned.bed"
output_file = "./test.bed"

peak = pybedtools.BedTool(peak_file)
annotation = pybedtools.BedTool(annotation_file)

intersection = peak.intersect(annotation, wo=True, s=True)

# Reformat the file generated by pybedtools to a simple Bed format
init_template=[0,"\t",1,"\t",2,"\t",3,"|",4,"|",5,"|",6,"|",7,"\t",8,"\t",9,"\t",10,"\t",11,"\t",12,"\t",13,
               "\t",14,"\t",15,"\t",16,"\t",17,"\tID=",18,";gene_id=",19,";gene_type=",20,";gene_status=",21,
               ";gene_name=",22,";level=",23,";havana_gene=",24]

final_template=[0,"\t",1,"\t",2,"\t",3,"|",4,"|",5,"|",6,"|",18,"|",20,"|",22,"\t",8,"\t",9]

print(intersection.head())

print("Post processing results")
reformat_table(
    input_file=intersection.fn,
    output_file=output_file,
    init_template=init_template,
    final_template=final_template,
    replace_internal_space='_',
    replace_null_val="-",
    keep_original_header = False,
    predicate = lambda v: v[12] == "gene"
)

head(output_file)

import pybedtools

def intersect_extract_genecodeID (annotation_file, peak_file, outdir):

    output_file = "{}/{}_{}.bed".format(outdir, file_basename(peak_file), file_basename(annotation_file))
    genecount_file = "{}/{}_{}_uniq-gene.csv".format(outdir, file_basename(peak_file), file_basename(annotation_file))
    site_file = "{}/{}_{}_uniq-sites.csv".format(outdir, file_basename(peak_file), file_basename(annotation_file))

    peak = pybedtools.BedTool(peak_file)
    annotation = pybedtools.BedTool(annotation_file)

    # Intersect the 2 files with pybedtools
    print("Intersecting {} with {}".format(file_basename(peak_file), file_basename(annotation_file)))
    intersection = peak.intersect(annotation, wo=True, s=True)

    # Reformat the file generated by pybedtools to a simple Bed format
   ...