I am working on the file "Soluciones.ipynb".
The first task is:
6.
A veces, es necesario en nuestro problema, tener que eliminar los elementos repetidos de una lista, dejando aquellos que solo aparezcan una sola vez. Es muy común, que muchos usuarios llamen a la función set para esta tarea, haciendo de la lista un conjunto sin elementos repetidos, ordenándolos y luego, el resultado de esto, volverlo a convertir en una lista. Esto, puede no estar mal del todo, pero puede ser que en el caso peor, puede que estemos haciendo un gasto inútil de memoria, tiempo y cálculos, para que, en el caso de que no haya elementos repetidos, sólo obtengamos una lista ordenada.
Es por ello, por lo que existe otra forma de hacerlo. Utilizando lo ya visto, obtén una lista sin elementos repetidos que mantengan el orden de la lista original. Para hacerlo aún más divertido, no uses más de 4 líneas.
Can you write Python code for it?

a = [1,1,1,2,5,3,4,8,5,8]
b = []
list(filter(lambda x: b.append(x) if not x in b else False, a))
print("Lista original:\t\t", a)
print("Lista sin repetidos:\t", b)


I am working on the file "Soluciones.ipynb".
The first task is:
6.
Haz una lista de la compra e imprime los siguientes elementos:

Penúltimo elemento
Del segundo al cuarto elemento
Los tres últimos
Todos!

Por último, elimina el tercer elemento de la lista usando la sentencia del
Can you write Python code for it?

lista_compra = ['Leche', 'Chocolate', 'Arroz', 'Macarrones']

print("Penúltimo elemento: ", lista_compra[-2])
print("Del segundo al cuarto elemento: ", lista_compra[1:5])
print("Los tres últimos elementos: ", lista_compra[-3:])
print("Todos: ", lista_compra)

del lista_compra[2]
print(lista_compra)


I am working on the file "Soluciones.ipynb".
The first task is:
7.
Crea una lista con todos los números pares del 0 al 10 en una única línea.
Can you write Python code for it?

# solución 1:
[x for x in range(10) if x%2==0]

# solución 2:
list(range(0,10,2))


I am working on the file "Soluciones.ipynb".
The first task is:
8.
Crea la siguiente matriz en una línea:
$$ M_{2 \times 3} = \left( \begin{matrix} 1 & 2 & 3 \
4 & 5 & 6 \end{matrix} \right)$$
Can you write Python code for it?

[[j for j in range(i*i, i*i+3)] for i in range(1,3)]


I am working on the file "Soluciones.ipynb".
The first task is:
Calcula la media y la desviación típica de la matriz.
Obtén el elemento mínimo y máximo de la matriz.
Calcula el determinante, la traza y la traspuesta de la matriz.
Calcula la descomposición en valores singulares de la matriz.
Calcula el valor de la suma de los elementos de la diagonal principal de la matriz.
Can you write Python code for it?

media = np.mean(n_array1)
desv_tipica = np.std(n_array1)

print("Media =", media, " y desv típica =", desv_tipica)

maximo = np.max(n_array1)
minimo = np.min(n_array1)

print("Máximo =", maximo, " y minimo =", minimo)

det = np.linalg.det(n_array1)
traza = np.trace(n_array1)
traspuesta = n_array1.T

U, S, V = np.linalg.svd(n_array1)
print(U)
print(S)
print(V)

result = np.diag(array1).sum()
print("Resultado: ", result)


I am working on the file "Soluciones.ipynb".
The first task is:
Multiplica array1 por $\frac{\pi}{4}$ y calcula el seno del array resultante.

Genera un nuevo array cuyo valor sea el doble del resultado anterior mas el vector array1.

Calcula la norma del vector resultante. Para ello, consulta la documentación para ver qué función realiza esta tarea, y ten en cuenta los parámetros que recibe.
Can you write Python code for it?

array2 = np.sin(array1 * np.pi/4)
array2

array3 = array2 * 2 + array1
array3

np.linalg.norm(array3)


I am working on the file "Soluciones.ipynb".
The first task is:
Nivel medio
1.
Ahora que hemos visto cómo crear arrays a partir de un objeto y otros para crear arrays con tipos prefijados, crea distintos arrays con las funciones anteriores para 1D, 2D y 3D e imprímelas por pantalla. Prueba a usar distintos tipos para ver cómo cambian los arrays. Si tienes dudas sobre cómo usarlos, puedes consultar la documentación oficial.
Can you write Python code for it?

import numpy as np

print(np.ones(5, dtype=np.int8))

print(np.random.random(5))

print(np.full(shape=(3,3), fill_value=4, dtype=np.int8))

print(np.arange(6))

print(np.linspace(start=1, stop=6, num=10))

print(np.eye(N=2))

print(np.identity(n=3, dtype=np.int8))


I am working on the file "Solution_Architecture_with_Ansible_and_Jupyter.ipynb".
The first task is:
Ansible Playbooks
One playbook with one task and three debug statments concerning the network adapater Ansible is using by default."
Ansible's debug module is used to
*  print the IPv4 address
*  print the MAC address
*  print the IPv4 and MAC address togehter on one line
Can you write Python code for it?

! nl playbooks/ch04-98_facts_ip-mac.yaml

! ansible-playbook --verbose --inventory=controlMachine/inventory.ini --become playbooks/ch04-98_facts_ip-mac.yaml


I am working on the file "Solution_Architecture_with_Ansible_and_Jupyter.ipynb".
The first task is:
Ansible's "setup" module; all it knows about a target machine
When Ansible's connects to a target machine it collects information as part of its setup.
The resulting data structure is collectively referred to as facts.
Can you write Python code for it?

oneSetup = !ansible aur[0] --inventory=controlMachine/inventory.ini --module-name=setup
print(f'{len(oneSetup):>34}  Metric of setup results; length of Jupyter reference to saved setup results.\n'
      f'{type(oneSetup)}  Type of Jupyter reference to save results.')

oneSetup  # View facts from the first target machine.


I am working on the file "Solution_Architecture_with_Ansible_and_Jupyter.ipynb".
The first task is:
Here Ansible runs a program on the remote machines.

connectivity is established
uptime for the remote machines may be of interest than the string "pong".
Can you write Python code for it?

!ansible aur --inventory=controlMachine/inventory.ini --module-name=command --args=uptime


I am working on the file "Solution_Architecture_with_Ansible_and_Jupyter.ipynb".
The first task is:
Report the target machines' date for rough idea of time synchrony.
From the control machine, Ansible concurrently accesses the target machines.
This example shows the three target machines are within a 100 milliseconds of each other.
Can you write Python code for it?

!ansible aur --inventory=controlMachine/inventory.ini --module-name=shell --args="date --rfc-3339=ns"


I am working on the file "Solution_Architecture_with_Ansible_and_Jupyter.ipynb".
The first task is:
Results from piping commands on the remote machines.
Three commands on the target host to yield an IPv4 address of a network device.
(CentOS uses the “Predictable Network Interface Names” convention.)
Can you write Python code for it?

!ansible aur --inventory=controlMachine/inventory.ini --module-name=shell --args="/usr/sbin/ip -4 addr show eno16777728 | sed -n '2p' | cut  -d ' ' -f 6"


I am working on the file "Solution_Architecture_with_Ansible_and_Jupyter.ipynb".
The first task is:
Review of control machine's Ansible configuration
Let the Jupyter notebook know where to find files.
Can you write Python code for it?


I am working on the file "Solution_Architecture_with_Ansible_and_Jupyter.ipynb".
The first task is:
View of the network, private, from /etc/hosts
(/etc/hosts display truncated to avoid line wrap)
Can you write Python code for it?


I am working on the file "Solution_Architecture_with_Ansible_and_Jupyter.ipynb".
The first task is:
is the control machine able to reach the remote machines?
Ansilble module, ping, replies with the string "ping" the control machine successfully connects with the host in the group aur.
Can you write Python code for it?

!ansible aur --inventory=controlMachine/inventory.ini --module-name=ping


I am working on the file "Solutions 4.ipynb".
The first task is:
But then there are also separate entries for each <li> in the list inside:
Can you write Python code for it?


I am working on the file "Solutions 4.ipynb".
The first task is:
Clean up dates and format them as datetimes
The formatting of the dates is not so great, so let's just clean that up.
Can you write Python code for it?


I am working on the file "Solutions 4.ipynb".
The first task is:
Extracting elements from the summary page
We will write a function to extract elements from a list of table rows (defined by <tr>).
Can you write Python code for it?

def extract_summary(trs):
    date_w_year, passengers, crew, fatalities, survivors = '', 0, 0, 0, 0
    registration, origins, destination = 'No data', 'No data', 'No data'

    for each_tr in trs:
        if each_tr.find('th', text = re.compile('Destination')) != None:
            try:
                destination = each_tr.td.text
            except:
                pass
        elif each_tr.find('th', text = re.compile('Date')) != None:
            try:
                date_w_year = each_tr.td.text
            except:
                pass
        elif each_tr.find('th', text = re.compile('Passengers')) != None:
            try:
                passengers = extract_numbers(each_tr.td.text, passengers)
            except:
                pass
        elif each_tr.find('th', text = re.compile('Crew')) != None:
            try:
                crew = extract_numbers(each_tr.td.text, passengers)
            except:
                pass
        elif each_tr.find('th', text = re.compile('Fatalities')) != None:
            try:
                fatalities = extract_numbers(each_tr.td.text, passengers)
            except:
                pass
        elif each_tr.find('th', text = re.compile('Survivors')) != None:
            try:
                survivors = extract_numbers(each_tr.td.text, passengers)
            except:
                pass
        elif each_tr.find('th', text = re.compile('Flight origin')) != None:
            try:
                origins = each_tr.td.text
            except:
                pass
        elif each_tr.find('th', text = re.compile('Registration')) != None:
            try:
                registration = each_tr.td.text
            except:
                pass
        else:
            pass

    return {'destination': destination,
            'date': date_w_year,
            'passengers': passengers,
            'crew': crew,
            'fatalities': fatalities,
            'survivors': survivors,
            'origins': origins,
            'registration': registration}


I am working on the file "Solutions 4.ipynb".
The first task is:
Handling special dates with multiple accidents
However, there are some elements in our list that contain another list of dates, which look something like this:
`
1950 Air France multiple Douglas DC-4 accidents:

June 12 – An Air France Douglas DC-4 (F-BBDE) on a flight from Saigon to Paris crashes in the Arabian Sea while on approach to Bahrain Airport, killing 46 of 52 on board.

June 14 – An Air France Douglas DC-4, F-BBDM, crashes in the Arabian Sea while on approach to Bahrain Airport, killing 40 of 53 on board. This aircraft was operating on the same flight route as F-BBDE.

`
We'll need to make sure we can handle these cases. Additionally, since there are <li> tags within this block, the <li>June 12...</li> and <li>June 14...</li> will appear again in our ResultSet. We should get rid of these duplicates.
Looking at the HTML above, the duplicated <li> entries won't have both links and date separators in them, so that's one way we can drop them. Then we'll save rows from the interior <li> tags, supplementing link information from the parent if necessary.
For example, the entry for September 11 is:
Can you write Python code for it?


I am working on the file "Solutions 4.ipynb".
The first task is:
Let's check that we did everything right for the weird cases by checking one of the bullets that had multiple crashes:
Can you write Python code for it?


I am working on the file "Solutions 4.ipynb".
The first task is:
Let's see the description:
Can you write Python code for it?


I am working on the file "Solutions 4.ipynb".
The first task is:
Let's test with the two URLs from earlier:
Can you write Python code for it?

test_collision_url = 'https://en.wikipedia.org/wiki/1922_Picardie_mid-air_collision'
summary_html = try_request(test_collision_url)

tables = summary_html.find_all('table', {"class" : "infobox vcard vevent"})
result_updated = extract_summaries(tables)

result_updated


I am working on the file "Solutions 4.ipynb".
The first task is:
Let's write a function that will take each <li> and extract the crash details (month, day, link, and text description) from each element. We should also make sure that we handle the cases where there are several crashes per <li>.
Can you write Python code for it?

def extract_details(html_fragment):
    # these lists may have one or more elements when returned
    bdates, blinks, bdescrips = [], [], []

    if html_fragment.find_all('li') == []:
        # Then there is only one crash for this bullet
        separator = get_date_separator(html_fragment)
        blinks.append(html_fragment.find('a').get('href'))
        bdates.append(html_fragment.text.split(separator)[0].strip())
        bdescrips.append(html_fragment.text.split(separator)[1].strip())

    else:
        # Then there are multiple crashes for this bullet
        for bullet in html_fragment.find_all('li'):

            # Dates might appear in current or parent

            separator = get_date_separator(bullet)
            if separator != None:
                bdates.append(bullet.text.split(separator)[0].strip())
                bdescrips.append(bullet.text.split(separator)[1].strip())
            else:
                parent_separator = get_date_separator(html_fragment)
                bdates.append(html_fragment.text.split(parent_separator)[0].strip())
                bdescrips.append(bullet.text.strip())

            # Relevant link might appear in current or parent

            if bullet.find('a') == None:
                blinks.append(html_fragment.find('a').get('href'))
            else:
                blinks.append(bullet.find('a').get('href'))

    return bdates, blinks, bdescrips

dates_month_day, links, descriptions = [], [], []
for each_li in result:
    if (' – ' in each_li.text or ' - ' in each_li.text) and each_li.find('a') != None:
        lis_dates, lis_links, lis_descrips = extract_details(each_li)
        dates_month_day += lis_dates
        links += lis_links
        descriptions += lis_descrips
    else:
        # If neither condition is true, then we hit duplicate or extra links
        # elsewhere in the page so we can skip these and throw them away
        continue


I am working on the file "Solutions 4.ipynb".
The first task is:
Looks like we can correctly extract a summary table with multiple aircraft in it.
Now let's try on a page that has multiple crashes in it on different days.
Can you write Python code for it?

test_multiple_dates_url = 'https://en.wikipedia.org/wiki/1950_Air_France_multiple_Douglas_DC-4_accidents'
summary_html = try_request(test_multiple_dates_url)

first_crash = 'June 12'
second_crash = 'June 14'

tables = summary_html.find_all('table', {"class" : "infobox vcard vevent"})
result_updated = extract_summaries(tables, first_crash)

result_updated

result_updated = extract_summaries(tables, second_crash)

result_updated

dates_w_year, passengers, crew, fatalities, survivors = [], [], [], [], []
registration, origins, destination = [], [], []

for num_row in range(len(links)):
    # Get HTML of detail page
    summary_html = try_request(base_url + links[num_row])

    # Get tables that are in these sidebars (mostly one, but sometimes multiple)
    tables = summary_html.find_all('table', {"class" : ["infobox", "vcard"]})

    # Extract data from summary HTML
    summary = extract_summaries(tables, dates_month_day[num_row])

    # Save the data for this page in our lists
    dates_w_year.append(summary['date'])
    passengers.append(summary['passengers'])
    crew.append(summary['crew'])
    fatalities.append(summary['fatalities'])
    survivors.append(summary['survivors'])
    origins.append(summary['origins'])
    registration.append(summary['registration'])
    destination.append(summary['destination'])

# Clean dates
cleaned_dates = [str(d).replace(',', '') for d in dates_w_year]
datescol = fix_dates(cleaned_dates)
dates_datetime = pd.to_datetime(datescol, format='%d %B %Y', errors='coerce')

# Save!
df_summary = pd.DataFrame({'date': dates_datetime, 'link': links, 'description': descriptions, 'passengers': passengers,
                       'crew': crew, 'fatalities': fatalities, 'survivors': survivors,
                       'registration': registration, 'flight origin': origins, 'destination': destination})
# save all this scraped stuff!
df_summary.to_csv('final_dataframe_summary.csv')


I am working on the file "Solutions 4.ipynb".
The first task is:
Optional: Doing part b while handling the special cases where there are multiple summaries per page
I told you to ignore this for the sake of simplifying this assignment, but it is the case that there are multiple summaries on a couple pages, for example:
https://en.wikipedia.org/wiki/1950_Air_France_multiple_Douglas_DC-4_accidents
These we can separate using the dates.
However, in addition to this, there are some cases where planes crash into one another. In these cases, the summaries are separated into two tables, one for each page, for example:
https://en.wikipedia.org/wiki/1922_Picardie_mid-air_collision
We could handle this in two ways: Sum the numbers for passengers, fatalities, etc. Or we could instead create two rows for these crashes.
We could handle both these cases by doing the following (pseudocode):
if there are multiple tables
        get the summary details for the appropriate date
else if there is one table
        get the summary details for each summary table and sum them (this is a collision)
We will create a new function extract_summaries() that will implement this approach.
Can you write Python code for it?

%pdb

def extract_summaries(tables, relevant_date):

    if len(tables) == 1:
        result = extract_single_table_summary(tables[0])
    else:
        result = extract_relevant_table_summary(tables, relevant_date)

    return {'destination': result['destination'],
            'date': result['date'],
            'passengers': result['passengers'],
            'crew': result['crew'],
            'fatalities': result['fatalities'],
            'survivors': result['survivors'],
            'origins': result['origins'],
            'registration': result['registration']}

def pick_out_table(tables, relevant_date):

    for table in tables:
        trs = table.find_all('tr')
        for each_tr in trs:
            if each_tr.find('th', text = re.compile('Date')) != None:

                # Clean and parse date
                date = each_tr.td.text.replace(',', '')
                parsed_date = parse_date_strings(date)

                if (parsed_date['month'] == relevant_date.split()[0]
                    and parsed_date['date'] == relevant_date.split()[1]):
                    return table

    return tables[0]

def extract_relevant_table_summary(tables, relevant_date):
    date_w_year, passengers, crew, fatalities, survivors = '', 0, 0, 0, 0
    registration, origins, destination = '', '', ''

    table = pick_out_table(tables, relevant_date)

    trs = table.find_all('tr')

    for each_tr in trs:
        if each_tr.find('th', text = re.compile('Destination')) != None:
            try:
                destination = each_tr.td.text
            except:
                pass
        elif each_tr.find('th', text = re.compile('Date')) != None:
            try:
                date_w_year = each_tr.td.text
            except:
                pass
        elif each_tr.find('th', text = re.compile('Passengers')) != None:
            try:
                passengers = extract_numbers(each_tr.td.text, passengers)
            except:
                pass
     ...


I am working on the file "Solutions 4.ipynb".
The first task is:
Question a: Setting up the dataframe
Can you write Python code for it?

base_url = "https://en.wikipedia.org"
index_ref = "/wiki/List_of_accidents_and_incidents_involving_commercial_aircraft"
index_html = urlopen(base_url + index_ref)
index = BeautifulSoup(index_html, "lxml")


I am working on the file "Solutions 4.ipynb".
The first task is:
Question b: Completing the dataframe with details
I'll rate limit my requests by using the function below:
Can you write Python code for it?

def try_request(url):
    html = urlopen(url)
    time.sleep(1)

    return BeautifulSoup(html, "lxml")


I am working on the file "Solutions 4.ipynb".
The first task is:
Question d: Which flight origin has the highest number of aviation incidents in the last 25 years?
It's 2016, so let's take accidents from 1991 and later and see which is the most common flight origin.
This crash is the first one to occur in 1991:
Can you write Python code for it?

df_full.date[672]

recent_incidents = df_full[673:]

recent_incidents['flight origin'].value_counts()[0:5]


I am working on the file "Solutions 4.ipynb".
The first task is:
Sanity check the lengths of each list:
Can you write Python code for it?


I am working on the file "Solutions 4.ipynb".
The first task is:
So the top 5 crashes, the number of fatalities and the flight origin was:
Can you write Python code for it?


I am working on the file "Solutions 4.ipynb".
The first task is:
This looks like exactly what we expected. Now let's proceed with clicking the links so we can add in the year. We have to click those links anyway to extract the additional crash details, so let's just grab the years from there.
Can you write Python code for it?

df.to_csv('crashes_question_starter.csv')

df[['description', 'link']].to_csv('crashes_no_extra_credit.csv')


I am working on the file "Solutions 4.ipynb".
The first task is:
Upon inspection of the text of each summary, we can see that there are some cases where in addition to (or instead of) an integer, there is some extraneous text or just a string like "all", "unknown", or "none". Let's handle these special cases and extract numbers in a function:
Can you write Python code for it?

def extract_numbers(td_text, passengers):
    """
    Function that handles table data rows to extract numbers.
    Handles special cases where there are strings like all, none, etc. in the text
    """

    number_regex = re.compile('\d+')
    all_regex = re.compile('ll')
    none_regex = re.compile('one')
    unknown_regex = re.compile('nknown')

    try:
        data_element = int(number_regex.findall(td_text)[0])
    except:
        if len(all_regex.findall(td_text)) >= 1:
            data_element = passengers
        elif len(none_regex.findall(td_text)) >= 1:
            data_element = 0
        elif len(unknown_regex.findall(td_text)) >= 1:
            data_element = 0
        else:
            data_element = 0

    return data_element


I am working on the file "Solutions 4.ipynb".
The first task is:
We can see now that our dates are nicely formatted and can create them as datetime objects:
Can you write Python code for it?

dates_datetime = pd.to_datetime(datescol, format='%d %B %Y', errors='coerce')

df_full['date'] = dates_datetime

df_full.head()

df_full = pd.DataFrame({'date': dates_datetime, 'link': links, 'description': descriptions, 'passengers': passengers,
                       'crew': crew, 'fatalities': fatalities, 'survivors': survivors,
                       'registration': registration, 'flight origin': origins, 'destination': destination})
# save all this scraped stuff!
df_full.to_csv('final_dataframe.csv')


I am working on the file "Solutions 4.ipynb".
The first task is:
Without de-duplication, Bergen Airport, Ninoy Aquino International Airport, and Domodedovo International Airport in Moscow had the highest number of aviation incidents.
Out of curiosity, let's do this for the entire dataset:
Can you write Python code for it?


I am working on the file "Solutions/01-Basic Calculations-solution.ipynb".
The first task is:
(2) 次のように各データの座標値 (x,y) のリストが与えられています。このデータの散布図を描いてください。
Can you write Python code for it?


I am working on the file "Solutions/02-Matrix and Vector-solution.ipynb".
The first task is:
(2) 逆行列を計算する関数 np.linalg.inv() を利用して、次の連立方程式の解を求めてください。
Can you write Python code for it?

w = np.array([[33],[59]])
m_inv = np.linalg.inv(m)
answer = np.dot(m_inv, w)
answer


I am working on the file "Solutions/02-Matrix and Vector-solution.ipynb".
The first task is:
(3) 次のarrayオブジェクトを「手早く」用意する方法を考えてください。
（ヒント：np.arange, reshape, T を利用します。）
Can you write Python code for it?


I am working on the file "Solutions/02-Matrix and Vector-solution.ipynb".
The first task is:
(4) arrayオブジェクトを受け取って、各要素を２倍する（*2 を演算する）関数 double を定義しなさい。また、(1)で定義した m を用いて、double(m) の結果を確認しなさい。
Can you write Python code for it?

def double(a):
    return a*2

double(m)


I am working on the file "Solutions/02-Matrix and Vector-solution.ipynb".
The first task is:
(5) double(m.tolist()) の結果を確認して、なぜそのような結果になったのかを考えてください。
（ヒント：tolist()メソッドは、arrayオブジェクトを通常の多次元リストに変換します。）
Can you write Python code for it?


I am working on the file "Solutions/02-Matrix and Vector-solution.ipynb".
The first task is:
練習問題
(1) 次の (2,2)行列 m と(2,1)行列（縦ベクトル）v をarrayオブジェクトとして定義してください。また、行列の積 w = mv を計算してください。
Can you write Python code for it?


I am working on the file "Solutions/03-Random Numbers-solution.ipynb".
The first task is:
(2) 2個のサイコロを振った結果を10回分用意します。次の例のように、1〜6の整数のペア（リスト）を10組含むarrayを生成して、変数 dice に保存してください。
Can you write Python code for it?


I am working on the file "Solutions/03-Random Numbers-solution.ipynb".
The first task is:
(4) 2個のサイコロの目の合計を1000回分用意して、2〜12のそれぞれの回数をヒストグラムに表示してください。
ヒント：オプション bins=11, range=(1.5, 12.5) を指定するときれいに描けます。
Can you write Python code for it?

dice = randint(1,7,[1000,2])
sums = [a+b for (a,b) in dice]
plt.hist(sums, bins=11, range=(1.5, 12.5))


I am working on the file "Solutions/03-Random Numbers-solution.ipynb".
The first task is:
練習問題
(1) 2個のサイコロを振った結果をシュミレーションします。次の例のように、1〜6の整数のペアを含むarrayを乱数で生成してください。
Can you write Python code for it?

from numpy.random import randint
randint(1,7,2)


I am working on the file "Solutions/04-Graph-solution.ipynb".
The first task is:
(2) -1≦x≦1 の範囲で、sin(2πx) と cos(2πx) のグラフを（上下2段に）別々に表示してください。軸ラベルや凡例などを好みで追加してください。
Can you write Python code for it?

data_x = np.linspace(-1,1,100)
data_sin = np.sin(2*np.pi*data_x)
data_cos = np.cos(2*np.pi*data_x)

fig = plt.figure(figsize=(6,4))

subplot = fig.add_subplot(2,1,1)
subplot.set_xlabel('x')
subplot.set_ylabel('y')
subplot.set_xlim(-1,1)
subplot.plot(data_x, data_sin, label=u'y=sin 2πx')
subplot.legend(loc=1)

subplot = fig.add_subplot(2,1,2)
subplot.set_xlabel('x')
subplot.set_ylabel('y')
subplot.set_xlim(-1,1)
subplot.plot(data_x, data_cos, label=u'y=cos 2πx')
subplot.legend(loc=1)


I am working on the file "Solutions/04-Graph-solution.ipynb".
The first task is:
(3) 次は、-10≦x≦10, -10≦y≦10（x, yは整数）の範囲について、x*x + y*y の値を保存した10x10行列（array）を用意して、ヒートマップを描くコードです。中間部分に必要なコードを追加して、実際にヒートマップを描いてください。
Can you write Python code for it?

data = np.array([[x**2 + y**2 for x in range(-10,11)] for y in range(-10,11)])
fig = plt.figure()
subplot = fig.add_subplot(1,1,1)
subplot.imshow(data, origin='lower', extent=(-10,10,-10,10), interpolation='nearest')


I am working on the file "Solutions/04-Graph-solution.ipynb".
The first task is:
(4) (3)において、imshowメソッドのオプション interpolation='nearest' を取ると結果がどのように変わるか確認しなさい。あるいは、オプション cmap=plt.cm.gray_r を付けるとどのように変わるか確認しなさい。
Can you write Python code for it?

data = np.array([[x**2 + y**2 for x in range(-10,11)] for y in range(-10,11)])
fig = plt.figure()
subplot = fig.add_subplot(1,1,1)
subplot.imshow(data, origin='lower', extent=(-10,10,-10,10))

data = np.array([[x**2 + y**2 for x in range(-10,11)] for y in range(-10,11)])
fig = plt.figure()
subplot = fig.add_subplot(1,1,1)
subplot.imshow(data, origin='lower', extent=(-10,10,-10,10), cmap=plt.cm.gray_r)


I am working on the file "Solutions/04-Graph-solution.ipynb".
The first task is:
左上を原点として目盛りを付けるには、extentオプションの指定を変更する必要があります。
Can you write Python code for it?

data = np.array([[x**2 + y**2 for x in range(0,11)] for y in range(0,11)])
fig = plt.figure()
subplot = fig.add_subplot(1,1,1)
subplot.imshow(data, extent=(0,10,10,0), interpolation='nearest')


I am working on the file "Solutions/04-Graph-solution.ipynb".
The first task is:
練習問題
(1) -1≦x≦1 の範囲で、sin(2πx) と cos(2πx) のグラフを重ねて表示してください。軸ラベルや凡例などを好みで追加してください。
Can you write Python code for it?

data_x = np.linspace(-1,1,100)
data_sin = np.sin(2*np.pi*data_x)
data_cos = np.cos(2*np.pi*data_x)

fig = plt.figure(figsize=(6,2))
subplot = fig.add_subplot(1,1,1)
subplot.set_xlabel('x')
subplot.set_ylabel('y')
subplot.set_xlim(-1,1)
subplot.plot(data_x, data_sin, label=u'y=sin 2πx')
subplot.plot(data_x, data_cos, label=u'y=cos 2πx')
subplot.legend(loc=1)


I am working on the file "Solutions/05-pandas DataFrame-01-Solution.ipynb".
The first task is:
(2) (1)の関数について、xの値と、対応する「y = sin(2πx) + 正規分布の乱数（平均0, 標準偏差0.3）」の値を列に持つDataFrameを返すように修正してください。（列の名前は、それぞれ 'x' および 'y' とします。）
例えば、次のような実装方法が考えられます。複数の方法を試してみてください。

列'x'と列'y'に対応するarrayオブジェクト data_x, data_y を用意した後、これらを列に持つDataFrameを定義する。（ディクショナリー形式のオプションで列を指定する。）
列'x'と列'y'に対応するarrayオブジェクト data_x, data_y を用意した後、空のDataFrameを作成して、df['x']=data_x のように列名指定でデータを追加する。
列'x'と列'y'を個別のSeriesオブジェクトとして作成した後、pd.concat()で結合する。
Can you write Python code for it?

def create_dataset1(num):
    data_x = np.linspace(0,1,num)
    data_y = np.sin(2*np.pi*data_x) + normal(loc=0, scale=0.3, size=num)
    df = DataFrame()
    df['x'] = data_x
    df['y'] = data_y
    return df

create_dataset1(10)

def create_dataset2(num):
    data_x = np.linspace(0,1,num)
    data_y = np.sin(2*np.pi*data_x) + normal(loc=0, scale=0.3, size=num)
    df = DataFrame({'x': data_x, 'y': data_y})
    return df

create_dataset2(10)

def create_dataset3(num):
    data_x = np.linspace(0,1,num)
    data_y = np.sin(2*np.pi*data_x) + normal(loc=0, scale=0.3, size=num)
    col_x = Series(data_x, name='x')
    col_y = Series(data_y, name='y')
    df = pd.concat([col_x, col_y], axis=1)
    return df

create_dataset3(10)


I am working on the file "Solutions/05-pandas DataFrame-01-Solution.ipynb".
The first task is:
練習問題
(1) 0≦x≦1の範囲を等分した num 個の点ついて、「sin(2πx) + 正規分布の乱数（標準偏差0.3）」を計算した array を返す関数 create_dataset(num) を作成してください。
Can you write Python code for it?

from numpy.random import normal

def create_dataset(num):
    data_x = np.linspace(0,1,num)
    data_y = np.sin(2*np.pi*data_x) + normal(loc=0, scale=0.3, size=num)
    return data_y

create_dataset(10)


I am working on the file "Solutions/07-pandas DataFrame-03-solution.ipynb".
The first task is:
(1) 所持金1000円からスタートして、所持金の50%を賭け続けた場合の所持金の変化をリスト trial に格納してください。（全部で100回繰り返します。）
Can you write Python code for it?

money = 1000
trial = []
for i in range(100):
    money = coin_game(money, int(money*0.5))
    trial.append(money)

trial[:10]


I am working on the file "Solutions/07-pandas DataFrame-03-solution.ipynb".
The first task is:
(2) (1)の処理を3回繰り返して、それぞれの結果を列（trial0, trial1, trial2）とするDataFrameを作成して、変数 df に格納してください。また、その結果をグラフ表示してください。
Can you write Python code for it?

df = DataFrame()
for t in range(3):
    money = 1000
    trial = []
    for i in range(100):
        money = coin_game(money, int(money*0.5))
        trial.append(money)
    df = pd.concat([df, Series(trial, name=('trial%d' % t))], axis=1)

df.plot(title='Coin Game')


I am working on the file "Solutions/07-pandas DataFrame-03-solution.ipynb".
The first task is:
(3) 関数 coin_game を修正して、1/2の確率で「掛け金の分だけ所持金が増加する」もしくは「掛け金の半分だけ所持金が減少」ようにしてください。修正後の関数名は stock_game とします。（これは、掛け金の分だけ株を購入したところ、1/2の確率が株価が倍、もしくは、半分になる状況をシュミレーションしています。）
Can you write Python code for it?

def stock_game(money, bet):
    coin = randint(2)
    if coin == 0:
        money += bet
    else:
        money -= int(bet/2)
    return money


I am working on the file "Solutions/07-pandas DataFrame-03-solution.ipynb".
The first task is:
(4) (3)の関数を用いて、(2)と同様のグラフを作成してください。
Can you write Python code for it?

result = DataFrame()
for t in range(3):
    money = 1000
    trial = []
    for i in range(100):
        money = stock_game(money, int(money*0.5))
        trial.append(money)
    result = pd.concat([result, Series(trial, name=('trial%d' % t))], axis=1)
result.plot(title='Stock Game')


I am working on the file "Solutions/07-pandas DataFrame-03-solution.ipynb".
The first task is:
次は、1000円の所持金において、100円を賭けた場合の結果を示します。
Can you write Python code for it?

money = 1000
money = coin_game(money, 100)
money


I am working on the file "Solutions/07-pandas DataFrame-03-solution.ipynb".
The first task is:
練習問題
次の関数 coin_game は、所持金と掛け金を引数に渡すと、1/2の確率で掛け金の分だけ所持金が増減した値が返ります。
Can you write Python code for it?

from numpy.random import randint
def coin_game(money, bet):
    coin = randint(2)
    if coin == 0:
        money += bet
    else:
        money -= bet
    return money


I am working on the file "Solutions/ProjectEuler.ipynb".
The first task is:
Problem 25
The Fibonacci sequence is defined by the recurrence relation:
Fn = Fn−1 + Fn−2, where F1 = 1 and F2 = 1.
Hence the first 12 terms will be:
F1 = 1
F2 = 1
F3 = 2
F4 = 3
F5 = 5
F6 = 8
F7 = 13
F8 = 21
F9 = 34
F10 = 55
F11 = 89
F12 = 144
The 12th term, F12, is the first term to contain three digits.
What is the index of the first term in the Fibonacci sequence to contain 1000 digits?
Can you write Python code for it?

def fibonacciSequence():
	old=1
	current=1
	new=0
	i=2
	while True:
		i+=1
		new=old+current
		if checkLength(new):
			break
		old=current
		current=new
	return i

def checkLength(n):
	if len(str(n))==1000:
		return True
	else:
		return False

print("Index of the first term in the Fibonacci sequence to contain 1000 digits: "+str(fibonacciSequence()))


I am working on the file "Solutions/ProjectEuler.ipynb".
The first task is:
Problem 26
A unit fraction contains 1 in the numerator. The decimal representation of the unit fractions with denominators 2 to 10 are given:
1/2 =   0.5
1/3 =   0.(3)
1/4 =   0.25
1/5 =   0.2
1/6 =   0.1(6)
1/7 =   0.(142857)
1/8 =   0.125
1/9 =   0.(1)
1/10    =   0.1
Where 0.1(6) means 0.166666..., and has a 1-digit recurring cycle. It can be seen that 1/7 has a 6-digit recurring cycle.
Find the value of d < 1000 for which 1/d contains the longest recurring cycle in its decimal fraction part.
Can you write Python code for it?

def cycleLength(n):
	res = 10
	j = 0
	while res != 10 or j < 1:
		res = (res % n) * 10
		j += 1
	return j

def Euler26():
	long = 0
	for i in range(2, 1000):
		if i%2 != 0 and i%5 != 0:
			length = cycleLength(i)
			if length > long:
				long = length
				res = i
	return res

print(Euler26())


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
As we will see below, even having two solutions is nothing but a numerical error.
Number of solutions as function of a
We can plot the (estimated) number of solution as a function of $a$, to start wit, thanks to the matplotlib.pyplot.hlines function:
Can you write Python code for it?

def plot_gap_points(gap_points, amin, amax):
    ys = set()
    for ym, yM in gap_points.keys():
        ys.add(ym)
        ys.add(yM)
    print(ys)
    xleft = dict()
    xright = dict()
    for (ym, yM), (xm, xM) in gap_points.items():
        xleft[ym] = xleft.get(ym, []) + [xm]
        xright[yM] = xright.get(yM, []) + [xM]
    for ym, yM in gap_points.keys():
        xleft[ym].sort()
        xright[yM].sort()
    print(xleft)
    print(xright)
    min_xleft = min(sum(list(xleft.values()), []))
    max_xright = min(sum(list(xright.values()), []))
    plt.figure()
    for y in ys:
        if y not in xleft and y in xright:
            for x in xright[y]:
                plt.hlines(y, x, amax)
        if y in xleft and min_xleft in xleft[y]:
            plt.hlines(y, amin, min_xleft)
            del xleft[y][0]
        #if y in xright and max_xright in xright[y]:
        #    plt.hlines(y, max_xright, amax)
        #    del xright[y][-1]
        if y in xleft and y in xright:
            for xmin, xmax in zip(xleft[y], xright[y]):
                plt.hlines(y, xmin, xmax)
    plt.xlabel("Value of $a$")
    plt.ylabel("Number of solution")
    plt.title(r"Number of solutions to $\exp(- a x^2) = x$, as function of $a$")

    return ys, xleft, min_xleft, xright, max_xright

ys, xleft, min_xleft, xright, max_xright = plot_gap_points(gap_points, amin, amax)


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
Asymptotic behaviors and approximations
We can try to approximate the solution for small $a$ or large $a$:

For small $a$, $W(2a) \simeq 2a - 4a^2$ so $x(a) \simeq 1 - a$.

For large $a$, we have this bound:
  $$ \forall x \geq \mathrm{e},{\displaystyle \ln(x)-\ln {\bigl (}\ln(x){\bigr )}+{\frac {\ln {\bigl (}\ln(x){\bigr )}}{2\ln(x)}}\leq W(x)\leq \ln(x)-\ln {\bigl (}\ln(x){\bigr )}+{\frac {e}{e-1}}{\frac {\ln {\bigl (}\ln(x){\bigr )}}{\ln(x)}}} $$
Can you write Python code for it?

e = np.exp(1)

def upper_bound(a):
    up_b = np.log(2*a) - np.log(np.log(2*a)) + (e / (e - 1)) * (np.log(np.log(2*a)) / np.log(2*a))
    return np.sqrt(up_b / (2*a))

def lower_bound(a):
    lo_b = np.log(2*a) - np.log(np.log(2*a)) + np.log(np.log(2*a)) / (2 * np.log(2*a))
    return np.sqrt(lo_b / (2*a))


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
First, let's have a look to its plot for some values of $a$:
Can you write Python code for it?

X = np.linspace(-2, 2, 2000)
for a in [0, -0.1, 0.1, -1, 1]:
    plt.plot(X, objective(X, a), 'o-', label=f"$a={a:.3g}$", markevery=50)
plt.legend()
plt.xlabel("$x$"); plt.ylabel("$y$")
plt.title(r"Function $\exp(- a x^2) - x$ for different $a$")
plt.show()


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
How many solutions for a given a ?
We can use this to try to find the threshold value for $a$ from $0$ to $2$ and from $2$ to $1$ solution:
Can you write Python code for it?

def thresholds(amin=-10, amax=10, delta=0.01):
    gap_points = dict()
    prev_a = amin
    prev_nb_sol = len(solutions(prev_a))
    for a in np.arange(amin, amax, delta):
        nb_sol = len(solutions(a))
        if nb_sol != prev_nb_sol:
            gap_points[(prev_nb_sol, nb_sol)] = (prev_a, a)
            prev_nb_sol = nb_sol
        prev_a = a
    return gap_points

thresholds(amin=-10, amax=10, delta=0.01)

thresholds(amin=-8, amax=1, delta=0.01)


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
I think having $3$ (or more) solutions is a numerical error.
Can you write Python code for it?

amin = -100
amax = 100
gap_points = thresholds(amin=amin, amax=amax, delta=0.1)
gap_points


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
Let's check that there is no solution for $a < 0$ too small.
Can you write Python code for it?


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
Plotting the function first
Can you write Python code for it?

def objective(x, a):
    return np.exp(- a * x**2) - x


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
Solving numerically?
We can start to try to use scipy.optimize.root to numerically solve this equation.
Can you write Python code for it?

def one_solution(a, x0=0, verb=False):
    sol = opt.root(objective, x0, args=(a,))
    if verb: print(sol)
    if sol.success:
        return sol.x
    else:
        raise ValueError(f"No solution was found for a = {a:.3g} (and starting at x0 = {x0:.3g}).")


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
This shows the numerical solution to the equation, and we will check below that the formal solution coincides.
Solving formally with the Lambert W function
Luckily, we can transform this equation to solve it with the Lambert $W$ function, defined as $W(x) = z \Leftrightarrow x = z \mathrm{e}^{z}$.
For more details, please see this page, or this article.
As for (almost) all the special function, we don't need to write it ourself: it is in scipy! scipy.special.lambertw
Can you write Python code for it?


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
We can check some values:
Can you write Python code for it?

for a in [0.5, 1, 2, 3, 4]:
    xa = formal_solution(a)
    assert np.isclose(exp(-a * xa**2), xa)
    print(f"a = {a:.3g} gives x(a) = {float(xa):.3g}")


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
We can plot all this.
Can you write Python code for it?

A = np.linspace(0, 20, 4000)
A1 = A[A <= 0.5]
A2 = A[A >= 1]
Ae = A[A >= e]

plt.plot(A, formal_solution(A), label="Solution", markevery=20)

plt.plot(A1, 1 - A1, 'b--', label=r"Tangent at $0$: $x(a) \simeq 1 - a$", markevery=20)

#plt.plot(A2, np.sqrt((np.log(2*A2) - np.log(np.log(2*A2)))/(2*A2)), 'g--', label=r"Asymptote at $+\infty$", markevery=20)
plt.plot(Ae, lower_bound(Ae), 'g--', label=r"Lower-bound for $a \geq e$", markevery=20)
plt.plot(Ae, upper_bound(Ae), 'c--', label=r"Upper-bound for $a \geq e$", markevery=20)

plt.legend()
plt.xlabel("Parameter $a$"); plt.ylabel(r"Solution $x(a) = \sqrt{\frac{W(2a)}{2a}}$")
plt.title(r"Solution to $\exp(- a x^2) = x$, as function of $a$")
plt.show()


I am working on the file "Solving_an_equation_and_the_Lambert_W_function.ipynb".
The first task is:
We can see that a solution to $\exp(-a x^2) = x$ has to be positive, as $\exp(-a x^2) > 0$ for any $x,a$.
We also check that if $a < 0$, $\exp(-a x^2) - x$ seems to always be positive, but if $a \geq 0$, it seems to have a unique root.
Let's zoom a little bit:
Can you write Python code for it?

X = np.linspace(0, 1.5, 2000)
for a in [0, -0.1, 0.1, -1, 1]:
    plt.plot(X, objective(X, a), 'o-', label=f"$a={a:.3g}$", markevery=50)
plt.legend()
plt.xlabel("$x$"); plt.ylabel("$y$")
plt.title(r"Function $\exp(- a x^2) - x$ for different $a$")
plt.show()


I am working on the file "Some Advanced Python.ipynb".
The first task is:
Lambda : Anonymous Functions
Lambda should not used for complex functions, however very handy for writing simple functions instead of defining funvtions
Can you write Python code for it?

my_func=lambda a,b:a*b

a=my_func(2,3)
a

for p in people:
    print( (lambda person:split_title_and_name(person))(p) )

list(map(split_title_and_name, people)) == list(map( (lambda person:person.split()[0] + person.split()[-1]),people))


I am working on the file "Sondeos Elecciones 2015.ipynb".
The first task is:
La extrapolación hacia adelante
Can you write Python code for it?

f, ax = pl.subplots(nrows=1, ncols=1, figsize=(15,8))
ax.xaxis_date()

gp = ml.GaussianProcess(delta)

for i in range(6):
    gp.fit(partidos[i], 0.05)
    predict, covariance = gp.predict(delta)
    ax.plot(fecha, partidos[i] - predict, '.', color=sn.xkcd_rgb[colors[i]], linewidth=2)
    print("{0} -> {1:3.1f}%".format(nombres[i], 100.0*np.std(partidos[i] - predict)))

f.autofmt_xdate()
ax.fmt_xdata = mdates.DateFormatter('%Y-%m-%d')
ax.xaxis.set_major_locator(mdates.MonthLocator())


I am working on the file "Sondeos Elecciones 2015.ipynb".
The first task is:
Leemos los sondeos obtenidos de la wikipedia (https://es.wikipedia.org/wiki/Anexo:Sondeos_de_intenci%C3%B3n_de_voto_para_las_elecciones_generales_de_Espa%C3%B1a_de_2015). Hacemos un poco de limpieza de los datos para dejar una fecha del sondeo y unificar los valores en tanto por uno. Generamos además un eje de tiempo.
Can you write Python code for it?

book = xlrd.open_workbook("sondeos.xlsx")
sh = book.sheet_by_index(0)

PP = []
PSOE = []
IU = []
UPyD = []
Podemos = []
Ciudadanos = []
fecha = []

mesEsp = ['ene', 'feb', 'mar', 'abr', 'may', 'jun', 'jul', 'ago', 'sep', 'oct', 'nov', 'dic']
mesEng = ['jan', 'feb', 'mar', 'apr', 'may', 'jun', 'jul', 'aug', 'sep', 'oct', 'nov', 'dec']

for rx in range(2,sh.nrows):
	row = sh.row_values(rx)

	res = row[2]

	out = bytes(res, 'utf-8')
	res = out.replace(b'\xe2\x80\x93', b'-').decode('utf-8').lower()

	for i in range(12):
		res = res.replace(mesEsp[i], mesEng[i])

	res = res.replace('.', '')

	guion = res.find('-')

	if (guion != -1):
		res = res[guion+1:]

	out = res.split(' ')

	if ((len(out) > 1) and (res.find(')') == -1) ):
		if (len(out) == 2):
			res = '1 '+res

		PP.append(row[3])
		PSOE.append(row[4])
		IU.append(row[5])
		UPyD.append(row[6])
		Podemos.append(row[16])
		Ciudadanos.append(row[17])

		fecha.append(dt.datetime.strptime(res, "%d %b %Y").date())

partidos = [PP, PSOE, IU, UPyD, Podemos, Ciudadanos]
nSondeos = len(PP)
# Now clean the lists to transform percentages to numbers
for partido in partidos:
	for i in range(nSondeos):
		if (not isinstance(partido[i], numbers.Number)):
			findPct = partido[i].find('%')
			if (findPct != -1):
				res = 0.01*float(partido[i][0:findPct].replace(',', '.'))
				partido[i] = res
			else:
				partido[i] = 0

PP = np.asarray(PP)[::-1]
PSOE = np.asarray(PSOE)[::-1]
IU = np.asarray(IU)[::-1]
UPyD = np.asarray(UPyD)[::-1]
Podemos = np.asarray(Podemos)[::-1]
Ciudadanos = np.asarray(Ciudadanos)[::-1]
fecha = np.asarray(fecha)[::-1]
delta = np.zeros(len(fecha))
for i in range(len(fecha)):
	delta[i] = (fecha[i]-fecha[0]).days


I am working on the file "Sorting/Sorting.ipynb".
The first task is:
From the above chart we can clearly see how Insertion Sort is faster for input size up to approximately 20.
Can you write Python code for it?

long_lists = [rand_list(max_len=1000) for _ in range(1000)]

test_lists = long_lists[:]
insertion_sort_times = [time_f(lambda: insertion_sort(x)) for x in test_lists]

test_lists = long_lists[:]
merge_sort_times = [time_f(lambda: merge_sort(x)) for x in test_lists]

list_sizes = [len(x) for x in long_lists]

%matplotlib inline
from matplotlib import pyplot as plt

isort = plt.scatter(list_sizes, insertion_sort_times, c='red')
msort = plt.scatter(list_sizes, merge_sort_times, c='blue')
plt.xlabel('n')
plt.ylabel('time (/s)')
plt.xlim(0)
plt.ylim(0)

plt.legend((isort, msort),
           ('Insertion sort', 'Merge sort'))
pass

very_long_lists = [rand_list(max_len=100000) for _ in range(100)]

merge_sort_times = [time_f(lambda: merge_sort(x)) for x in very_long_lists]
list_sizes = [len(x) for x in very_long_lists]

%matplotlib inline
from matplotlib import pyplot as plt

merge_sort_data = plt.scatter(list_sizes, merge_sort_times, c='blue')
plt.xlabel('n')
plt.ylabel('time (/s)')
plt.xlim(0)
plt.ylim(0)

plt.legend((merge_sort_data,),
           ('merge_sort',))


I am working on the file "Sorting/Sorting.ipynb".
The first task is:
Use this test to confirm your implementation is correct.
Can you write Python code for it?


I am working on the file "SoundscapeComponentAnalysis.ipynb".
The first task is:

Example audio

Load an example audio file from the 'sounds' directory, 44.1kHz, stereo, 60 seconds duration.
Can you write Python code for it?

sound_path = 'sounds'
sounds = os.listdir(sound_path)
print "sounds:", sounds


I am working on the file "SoundscapeComponentAnalysis.ipynb".
The first task is:

Spectrum Analysis Parameters

Inspect the soundfile by loading it (using wavread) and printing some useful parameters.
A window size of 4096 with a hop of 1024 translates to 92ms and 23ms respectively for an audio samplerate of 44100Hz
Can you write Python code for it?

N=4096; H=N/4
x,sr,fmt = wavread(os.path.join(sound_path,sounds[0]))
print "sample_rate:", sr, "(Hz), fft size:", (1000*N)/sr, "(ms), hop size:", (1000*H)/sr, "(ms)"


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Also pretty good, but not the best so far.  This algorithm outputs the importances of the features, which shows that $u-g$ is the most important with $r-i$ being next.
The main parameter for boosting is the depth of the underlying trees.  Let's make that a parameter and do a grid search over a range of depths.  The training set should always improve with greater depth, but the test set will plateau.  That's the value that we want for the depth.
Can you write Python code for it?

# Depth-optimized Boosting
depth = np.arange(1, 10)
score_test = np.zeros(len(depth))
score_train = np.zeros(len(depth))
i_best = 0
z_fit_best = None

# Loop over the depths
for i, d in enumerate(depth):
    clf = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, max_depth=d, random_state=42)
    clf.fit(Xtrain, ytrain)

    ypredtrain = clf.predict(Xtrain)
    # We need some way to decide what is best
    # Try using the f1_score and comparing the values for the different depths
    score_train[i] = f1_score(ytrain, ypredtrain)

    ypredtest = clf.predict(Xtest)
    score_test[i] = f1_score(ytest, ypredtest)

    if score_test[i] >= score_test[i_best]:
        i_best = i
        ybest = ypredtest

best_depth = depth[i_best]

# Give the best depth and the corresponding classification report.
print best_depth
print(classification_report(ytest, ybest, target_names=['QSOs', 'stars']))

# Plot the f1 score vs. depth
fig = plt.figure(figsize=(6, 6))
plt.plot(depth,score_test,label='test')
plt.plot(depth,score_train,label='train')
plt.xlabel("depth")
plt.ylabel("score")
plt.legend()
plt.show()


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Basically the same with 7 neighbors instead of the default of 5.  Scaled seems to be somewhat better than not. Try cross-validation.
Can you write Python code for it?

from sklearn.ensemble import BaggingClassifier
from sklearn.neighbors import KNeighborsClassifier
N = 5
clf = BaggingClassifier(KNeighborsClassifier(n_neighbors=7), max_samples=0.5, max_features=1.0, random_state=42, n_jobs=-1)
ypredCVBAG = cross_val_predict(clf, XS, y, cv=N)
print(classification_report(y, ypredCVBAG, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Comparison with KDE Results
Assuming class1 is stars and class2 is quasars:

  689365 true stars.  4193 true quasars.

  690189 points were labeled as stars.

  3369 points were labeled as quasars.

  689273 out of 689365 stars were labeled as stars (99.99%).

  92 out of 689365 stars were labeled as quasars (0.01335%).

  916 out of 4193 quasars were labeled as stars (21.85%).

  3277 out of 4193 quasars were labeled as quasars (78.15%).

Overall Accuracy: 99.85%

Weighted Accuracy: 89.07%

Completeness: 78.15%

Efficiency: 97.27%

Rating: 0.7602

This is about as good as the KDE results (at least for self classification), which got 78% completeness with 97% efficiency (where some of those stars classified as quasars are also likely $z\sim3.5$).  The combination of these suggests that the candidate list that John is using should has 3% or less contamination.  We can check that by also testing the photo-z's.
Make a few plots illustrating that not all of the contamination is from stars (but is instead from $z\lesssim3.5$).
Can you write Python code for it?

%matplotlib inline
fig = plt.figure(figsize=(12, 8))
fig.subplots_adjust(hspace=0.05)
ax1 = fig.add_subplot(211)
ax1.xaxis.set_major_formatter(plt.NullFormatter())
plt.scatter(datass['zspec'],datass['imag'],label='ss',edgecolor='None',color='m')
plt.scatter(dataqq['zspec'],dataqq['imag'],label='qq',edgecolor='None',color='b',alpha=0.5)
plt.scatter(dataqs['zspec'],dataqs['imag'],label='qs',edgecolor='None',color='k',alpha=0.5)
plt.scatter(datasq['zspec'],datasq['imag'],label='sq',edgecolor='None',color='g',alpha=0.5)
plt.legend(scatterpoints=1)
plt.xlim([-0.1,6.5])
plt.xlabel('zspec')
plt.ylabel('imag')

ax2 = fig.add_subplot(212, sharex=ax1)
plt.scatter(datass['zspec'],datass['gr'],label='ss',edgecolor='None',color='m')
plt.scatter(dataqq['zspec'],dataqq['gr'],label='qq',edgecolor='None',color='b',alpha=0.5)
plt.scatter(dataqs['zspec'],dataqs['gr'],label='qs',edgecolor='None',color='k',alpha=0.5)
plt.scatter(datasq['zspec'],datasq['gr'],label='sq',edgecolor='None',color='g',alpha=0.5)
plt.legend(scatterpoints=1)
plt.xlim([-0.1,6.5])
plt.xlabel('zspec')
plt.ylabel('gr')

plt.show()

fig = plt.figure(figsize=(12, 4))
#fig.subplots_adjust(vspace=0.05)
ax1 = fig.add_subplot(131)
plt.scatter(datass['ug'],datass['s1s2'],label='ss',edgecolor='None',color='m')
plt.scatter(dataqq['ug'],dataqq['s1s2'],label='qq',edgecolor='None',color='b',alpha=0.5)
plt.scatter(dataqs['ug'],dataqs['s1s2'],label='qs',edgecolor='None',color='k',alpha=0.5)
plt.scatter(datasq['ug'],datasq['s1s2'],label='sq',edgecolor='None',color='g',alpha=0.5)
plt.legend(scatterpoints=1)
plt.xlim([-4,8])
plt.ylim([-1.5,1.5])
plt.xlabel('u-g')
plt.ylabel('ch1-ch2')

ax2 =...


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Final result is 79% completeness and 8% contamination.
Should output the classification vector and do some exploring in TOPCAT with it.
Create a new Table column for data filled with the predicted $y$ values.  Then write it out as FITS.
Can you write Python code for it?

data['ypred'] = ypredCV

# Input file output with results of self-classification with RF
data.write('GTR-ADM-QSO-ir-testhighz_findbw_lup_2016_starclean_class.fits', format='fits')


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Make a redshift histogram to see how well that works.  Compare this to the results for RF.
Can you write Python code for it?

qq = ((data['labels']==0) & (data['ypred']==0))
ss = ((data['labels']==1) & (data['ypred']==1))
qs = ((data['labels']==0) & (data['ypred']==1))
sq = ((data['labels']==1) & (data['ypred']==0))
dataqq = data[qq]
datass = data[ss]
dataqs = data[qs]
datasq = data[sq]

print len(dataqq), "quasars selected as quasars"
print len(dataqs), "quasars selected as stars"
print len(datasq), "stars selected as quasars"

fig = plt.figure(figsize=(6, 6))
bins=np.linspace(0,5.5,100)
plt.hist(dataqq['zspec'],bins=bins,label='qq',color='b',alpha=0.5)
plt.hist(dataqs['zspec'],bins=bins,label='qs',color='r',alpha=0.5)
plt.hist(datasq['zspec'],bins=bins,label='sq',color='g',alpha=0.5)
#plt.xlim([-4,8])
#plt.ylim([-1.5,1.5])
plt.legend()
plt.xlabel('zspec')
plt.ylabel('N')


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Make the same histograms as above.
Can you write Python code for it?


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Neural Network Classification
Let's start by trying a neural network.  (More because that's the last thing that I taught than anything else.)  Set random_state so that this is repeatable.
Can you write Python code for it?

# Neural Network
from sklearn.neural_network import MLPClassifier
# Instantiate the classifier
nn = MLPClassifier(solver='lbfgs', alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1)
# Train the classifier with the training subset
nn.fit(Xtrain, ytrain)

# Now predict the labels for the test data
# ypred = clf.predict(Xtest)
# Added timing function for comparison with other methods.
ypred = nn.predict(Xtest)

# The classification report gives:
# efficiency = precision
# completeness = recall
# while the f1-score is a combination of the two
print(classification_report(ytest, ypred, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Now do the same with the scaled data.
Can you write Python code for it?

# Bagging
from sklearn.ensemble import BaggingClassifier
from sklearn.neighbors import KNeighborsClassifier
bagging = BaggingClassifier(KNeighborsClassifier(n_neighbors=7), max_samples=0.5, max_features=1.0, random_state=42, n_jobs=-1)
bagging.fit(XStrain, yStrain)
ypredBag = bagging.predict(XStest)

print(classification_report(ytest, ypredBag, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
OK, so something like 80 stars are making it into the sample.  Keep that in mind for comparison with the other algorithms.

Random Forest Classification
Now we'll try a DecisionTree, a RandomForest, and an ExtraTrees classifier
Note that n_jobs=-1 is supposed to allow it to use multiple processesors if it can, but I'm honestly not sure how that works (and also not convinced that it isn't causing problems as sometimes when I use it I get a warning that it is setting n_jobs=1 instead.
Can you write Python code for it?

# Random Forests, etc.
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import ExtraTreesClassifier
from sklearn.tree import DecisionTreeClassifier

clfDTC = DecisionTreeClassifier(max_depth=None, min_samples_split=2, random_state=42)
clfRFC = RandomForestClassifier(n_estimators=10, max_depth=None, min_samples_split=2, random_state=42, n_jobs=-1)
clfETC = ExtraTreesClassifier(n_estimators=10, max_depth=None, min_samples_split=2, random_state=42)

clfDTC.fit(Xtrain, ytrain)
clfRFC.fit(Xtrain, ytrain)
clfETC.fit(Xtrain, ytrain)

ypredDTC = clfDTC.predict(Xtest)
ypredRFC = clfRFC.predict(Xtest)
ypredETC = clfETC.predict(Xtest)

print clfDTC.feature_importances_
print clfRFC.feature_importances_
print clfETC.feature_importances_

print(classification_report(ytest, ypredDTC, target_names=['QSOs', 'stars']))
print(classification_report(ytest, ypredRFC, target_names=['QSOs', 'stars']))
print(classification_report(ytest, ypredETC, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Overall: 78% Completness and 93% Efficiency.  Good!  Now make a redshift histogram to compare to the one above.
Can you write Python code for it?

data['ypred'] = ypredCVBAG

qq = ((data['labels']==0) & (data['ypred']==0))
ss = ((data['labels']==1) & (data['ypred']==1))
qs = ((data['labels']==0) & (data['ypred']==1))
sq = ((data['labels']==1) & (data['ypred']==0))
dataqq = data[qq]
datass = data[ss]
dataqs = data[qs]
datasq = data[sq]

print len(dataqq), "quasars selected as quasars"
print len(dataqs), "quasars selected as stars"
print len(datasq), "stars selected as quasars"

fig = plt.figure(figsize=(6, 6))
bins=np.linspace(0,5.5,100)
plt.hist(dataqq['zspec'],bins=bins,label='qq',color='b',alpha=0.5)
plt.hist(dataqs['zspec'],bins=bins,label='qs',color='r',alpha=0.5)
plt.hist(datasq['zspec'],bins=bins,label='sq',color='g',alpha=0.5)
#plt.xlim([-4,8])
#plt.ylim([-1.5,1.5])
plt.legend()
plt.xlabel('zspec')
plt.ylabel('N')


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Pretty good.  72% completeness and 95% efficiency.
Do it again with scaled data to see if that makes any difference.
It doesn't seem to for colors alone, but might for other attributes?
Can you write Python code for it?

svm = SVC(random_state=42)
svm.fit(XStrain, yStrain)

ySpredSVM = svm.predict(XStest)
print(classification_report(yStest, ySpredSVM, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
So far a depth=17 RandomForest is giving the best results:
78% Completeness
7% Contamination
Though we might be able to optimize more by tweaking the other paramters.
Can you write Python code for it?

# Try increasing the number of estimators
clf = RandomForestClassifier(n_estimators=20, max_depth=15, min_samples_split=2, random_state=42, n_jobs=-1)
clf.fit(Xtrain, ytrain)
ypred = clf.predict(Xtest)

print(classification_report(ytest, ypred, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Somehow this isn't quite as good, so stick with 10 estimators.
But first need to produce classifications for all using cross-validation and the parameters above.
Can you write Python code for it?

from sklearn.model_selection import cross_val_predict
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report
clf = RandomForestClassifier(n_estimators=10, max_depth=15, min_samples_split=2, n_jobs=-1, random_state=42)
N=5
ypredCV = cross_val_predict(clf, X, y, cv=N)

print(classification_report(y, ypredCV, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
This actually has a lot less stellar contamination than the RF results below and the SVM results above.  Should consider using this instead (or in addition) in order to test the stellar contamination.

Boosting
Now try a boosting-based classifier.
Can you write Python code for it?

# Boosting
from sklearn.metrics import f1_score
from sklearn.ensemble import GradientBoostingClassifier
boost = GradientBoostingClassifier(n_estimators=200, learning_rate=0.1, max_depth=3, random_state=42)
boost.fit(Xtrain, ytrain)
ypredBoost = boost.predict(Xtest)

print f1_score(ytest, ypredBoost)
print(classification_report(ytest, ypredBoost, target_names=['QSOs', 'stars']))
print boost.feature_importances_


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
This is actually worse that for both SVM and Bagging.  But it is what I decided to go with for the first comparison with KDE.  We'll come back and look at this more in the next notebook.  However, it is clear that we might want to use something else if contamination is the main thing that we are worried about.
What is interesting is that about half of the "stars" are quasars with $\lesssim3.5$.
So, the contamination is more like 4% than 8%.
Similarly, most of the failed quasars are at $z\gtrsim3$, which isn't all that surprising.
Can you write Python code for it?


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
This suggests that a depth of 3 is optimal.

RandomForests were better than Boosting, so let's try optimizing the depth for RandomForests.
Can you write Python code for it?

# Depth-optimized Random Forest
depth = np.arange(10, 21)
score_test = np.zeros(len(depth))
score_train = np.zeros(len(depth))
i_best = 0
z_fit_best = None

for i, d in enumerate(depth):
    clf = RandomForestClassifier(n_estimators=10, max_depth=d, min_samples_split=2, random_state=42, n_jobs=-1)
    clf.fit(Xtrain, ytrain)

    ypredtrain = clf.predict(Xtrain)
    score_train[i] = f1_score(ytrain, ypredtrain)

    ypredtest = clf.predict(Xtest)
    score_test[i] = f1_score(ytest, ypredtest)

    if score_test[i] >= score_test[i_best]:
        i_best = i
        ybest = ypredtest

best_depth = depth[i_best]
print best_depth
print(classification_report(ytest, ybest, target_names=['QSOs', 'stars']))

fig = plt.figure(figsize=(6, 6))
plt.plot(depth,score_test,label='test')
plt.plot(depth,score_train,label='train')
plt.xlabel("depth")
plt.ylabel("score")
plt.legend()
plt.show()


I am working on the file "SpIESHighzCandidateSelection.ipynb".
The first task is:
Timing test
One of the reasons that I settled on RF is shown by the timing analysis below.  While the bagging analysis works well, it takes forever.  For just 173,390 sources bagging takes 32.8s as compared to 129ms for RF and SVM takes 7.45s.
So the prediction for classifying 50 million sources is:

RF 37 seconds

SVM 35.8 minutes

BAG 2.62 hours
Can you write Python code for it?

%timeit -n1 ypred = nn.predict(Xtest)

%timeit -n1 ypredRFC = clfRFC.predict(Xtest)

%timeit -n1 ypredBag = bagging.predict(XStest)

%timeit -n1 ySpredSVM = svm.predict(XStest)


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
Better with scaled data.   Not as complete as SVM, but more efficient.  Try changing number of neighbors.
Can you write Python code for it?

# Bagging
from sklearn.ensemble import BaggingClassifier
from sklearn.neighbors import KNeighborsClassifier
bagging = BaggingClassifier(KNeighborsClassifier(n_neighbors=7), max_samples=0.5, max_features=1.0, random_state=42, n_jobs=-1)
bagging.fit(XStrain, ytrain)
ypredBag3 = bagging.predict(XStest)

print(classification_report(ytest, ypredBag3, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
Don't forget that many of the "stars" are quasars with $\lesssim3.5$.
The bagged KNN looks to be pretty clean of stars even though the contamination rate
quoted above is high.  See the redshift histograms below.
Similarly, most of the failed quasars are at $z\gtrsim3$, which isn't all that surprising.
This is about as good as the KDE results (at least for self classification), which got 78% completeness with 97% efficiency (where some of those stars classified as quasars are also likely $z\sim3.5$).  The combination of these suggests that the candidate list that John is using should has 3% or less contamination.  We can check that by also testing the photo-z's.
Make a few plots illustrating that not all of the contamination is from stars (but is instead from $z\lesssim3.5$).
Can you write Python code for it?

fig = plt.figure(figsize=(12, 4))
#fig.subplots_adjust(vspace=0.05)
ax1 = fig.add_subplot(131)
plt.scatter(datass['ug'],datass['s1s2'],label='ss',edgecolor='None',color='m')
plt.scatter(dataqq['ug'],dataqq['s1s2'],label='qq',edgecolor='None',color='b',alpha=0.5)
plt.scatter(dataqs['ug'],dataqs['s1s2'],label='qs',edgecolor='None',color='k',alpha=0.5)
plt.scatter(datasq['ug'],datasq['s1s2'],label='sq',edgecolor='None',color='g',alpha=0.5)
plt.legend(scatterpoints=1)
plt.xlim([-4,8])
plt.ylim([-1.5,1.5])
plt.xlabel('u-g')
plt.ylabel('ch1-ch2')

ax2 = fig.add_subplot(132)
ax2.yaxis.set_major_formatter(plt.NullFormatter())
plt.scatter(datass2['ri'],datass2['s1s2'],label='ss',edgecolor='None',color='m')
plt.scatter(dataqq2['ri'],dataqq2['s1s2'],label='qq',edgecolor='None',color='b',alpha=0.5)
plt.scatter(dataqs2['ri'],dataqs2['s1s2'],label='qs',edgecolor='None',color='k',alpha=0.5)
plt.scatter(datasq2['ri'],datasq2['s1s2'],label='sq',edgecolor='None',color='g',alpha=0.5)
plt.legend(scatterpoints=1)
plt.xlim([-2,4.5])
plt.ylim([-1.5,1.5])
plt.xlabel('r-i')
plt.ylabel('ch1-ch2')

ax3 = fig.add_subplot(133)
ax3.yaxis.set_major_formatter(plt.NullFormatter())
plt.scatter(datass2['zs1'],datass2['s1s2'],label='ss',edgecolor='None',color='m')
plt.scatter(dataqq2['zs1'],dataqq2['s1s2'],label='qq',edgecolor='None',color='b',alpha=0.5)
plt.scatter(dataqs2['zs1'],dataqs2['s1s2'],label='qs',edgecolor='None',color='k',alpha=0.5)
plt.scatter(datasq2['zs1'],datasq2['s1s2'],label='sq',edgecolor='None',color='g',alpha=0.5)
plt.legend(scatterpoints=1)
plt.xlim([-3,6])
plt.ylim([-1.5,1.5])
plt.xlabel('z-ch1')
plt.ylabel('ch1-ch2')

plt.show()


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
Efficiency the best so far, but relatively low completeness.  Try scaled.
Can you write Python code for it?

bagging.fit(XStrain, ytrain)
ypredBag2 = bagging.predict(XStest)

print(classification_report(ytest, ypredBag2, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
Even with scaled data, none of these are as good as SVM.

Bagging
Now we'll try a bagging classifier, based on K Nearest Neighbors.
Can you write Python code for it?

# Bagging
from sklearn.ensemble import BaggingClassifier
from sklearn.neighbors import KNeighborsClassifier
bagging = BaggingClassifier(KNeighborsClassifier(), max_samples=0.5, max_features=1.0, random_state=42, n_jobs=-1)
bagging.fit(Xtrain, ytrain)
ypredBag = bagging.predict(Xtest)

print(classification_report(ytest, ypredBag, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
None of these are as good as SVM
Can you write Python code for it?

# Now scaled

clfDTC.fit(XStrain, ytrain)
clfRFC.fit(XStrain, ytrain)
clfETC.fit(XStrain, ytrain)

ypredDTC2 = clfDTC.predict(XStest)
ypredRFC2 = clfRFC.predict(XStest)
ypredETC2 = clfETC.predict(XStest)

print clfDTC.feature_importances_
print clfRFC.feature_importances_
print clfETC.feature_importances_

print(classification_report(ytest, ypredDTC2, target_names=['QSOs', 'stars']))
print(classification_report(ytest, ypredRFC2, target_names=['QSOs', 'stars']))
print(classification_report(ytest, ypredETC2, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
Note that I think for this classifier a "good" score is lower.  So, the best number of neighbors is >~9.
Can you write Python code for it?

# Bagging study
from sklearn.ensemble import BaggingClassifier
from sklearn.neighbors import KNeighborsClassifier
bagging = BaggingClassifier(KNeighborsClassifier(n_neighbors=9), max_samples=0.5, max_features=1.0, n_jobs=-1, random_state=42)
bagging.fit(XStrain, ytrain)
ypredBagKNN = bagging.predict(XStest)

print(classification_report(ytest, ypredBagKNN, target_names=['QSOs', 'stars']))

# Bagging with RF
from sklearn.ensemble import BaggingClassifier
bagging = BaggingClassifier(RandomForestClassifier(n_estimators=10, max_depth=None, min_samples_split=2), max_samples=0.5, max_features=1.0, n_jobs=-1, random_state=42)
bagging.fit(XStrain, ytrain)
ypredBag = bagging.predict(XStest)

print(classification_report(ytest, ypredBag, target_names=['QSOs', 'stars']))

# Bagging with SVM
from sklearn.ensemble import BaggingClassifier
bagging = BaggingClassifier(SVC(), max_samples=0.5, max_features=1.0, n_jobs=-1, random_state=42)
bagging.fit(XStrain, ytrain)
ypredBag = bagging.predict(XStest)

print(classification_report(ytest, ypredBag, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
OK, SVM alone gets highest completeness and bagging with KNN gets highest efficiency.  Do CV for both to classify all and look at the results.
Can you write Python code for it?

clf = SVC(random_state=42)
N=5
ypredCVSVM = cross_val_predict(clf, XS, y, cv=N)

print(classification_report(y, ypredCVSVM, target_names=['QSOs', 'stars']))

clf = BaggingClassifier(KNeighborsClassifier(n_neighbors=7), max_samples=0.5, max_features=1.0, n_jobs=-1, random_state=42)
N=5
ypredCVbagKNN = cross_val_predict(clf, XS, y, cv=N)

print(classification_report(y, ypredCVbagKNN, target_names=['QSOs', 'stars']))

data['ypredSVM'] = ypredCVSVM
data['ypredBAGKNN'] = ypredCVbagKNN

qq = ((data['labels']==0) & (data['ypredSVM']==0))
ss = ((data['labels']==1) & (data['ypredSVM']==1))
qs = ((data['labels']==0) & (data['ypredSVM']==1))
sq = ((data['labels']==1) & (data['ypredSVM']==0))
dataqq = data[qq]
datass = data[ss]
dataqs = data[qs]
datasq = data[sq]

print len(dataqq), "quasars selected as quasars"
print len(dataqs), "quasars selected as stars"
print len(datasq), "stars selected as quasars"

qq2 = ((data['labels']==0) & (data['ypredBAGKNN']==0))
ss2 = ((data['labels']==1) & (data['ypredBAGKNN']==1))
qs2 = ((data['labels']==0) & (data['ypredBAGKNN']==1))
sq2 = ((data['labels']==1) & (data['ypredBAGKNN']==0))
dataqq2 = data[qq2]
datass2 = data[ss2]
dataqs2 = data[qs2]
datasq2 = data[sq2]

print len(dataqq2), "quasars selected as quasars"
print len(dataqs2), "quasars selected as stars"
print len(datasq2), "stars selected as quasars"

data.write('GTR-ADM-QSO-ir-testhighz_findbw_lup_2016_starclean_class2.fits', format='fits')


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
Pretty good!  73% completeness and 95% efficiency.  But not as good as colors only.
Can you write Python code for it?

from sklearn.svm import SVC
svm = SVC(random_state=42)
svm.fit(XStrain, ytrain)

ypredSVM2 = svm.predict(XStest)
print(classification_report(ytest, ypredSVM2, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
Random Forest Classification
Now we'll try a DecisionTree, a RandomForest, and an ExtraTrees classifier
Can you write Python code for it?


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
SVM Classification
Can you write Python code for it?

from sklearn.svm import SVC
svm = SVC(random_state=42)
svm.fit(Xtrain, ytrain)

ypredSVM = svm.predict(Xtest)
print(classification_report(ytest, ypredSVM, target_names=['QSOs', 'stars']))

print(classification_report(ytest, ypredSVM, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
Scaled results are better, however.  81% completeness and 95% efficiency.  But would like even better efficiency if possible.
Play with some of the options.  Linear and poly (with other defaults) are bad.  Sigmoid is terrible!
Can you write Python code for it?

from sklearn.svm import SVC
svm = SVC(kernel='sigmoid', random_state=42)
svm.fit(XStrain, ytrain)

ypredSVM2 = svm.predict(XStest)
print(classification_report(ytest, ypredSVM2, target_names=['QSOs', 'stars']))


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
Star contam appears to be lower than for RF in SpIESHighzCandidateSelection
Histogram of Bagging selection.
Can you write Python code for it?

fig = plt.figure(figsize=(6, 6))
bins=np.linspace(0,5.5,100)
plt.hist(dataqq2['zspec'],bins=bins,label='qq',color='b',alpha=0.5)
plt.hist(dataqs2['zspec'],bins=bins,label='qs',color='r',alpha=0.5)
plt.hist(datasq2['zspec'],bins=bins,label='sq',color='g',alpha=0.5)
#plt.xlim([-4,8])
#plt.ylim([-1.5,1.5])
plt.legend()
plt.xlabel('zspec')
plt.ylabel('N')


I am working on the file "SpIESHighzCandidateSelection2.ipynb".
The first task is:
Yes!  This is the best so far!.  97% efficiency.  Still not 81% complete, but pretty good.

Boosting
Now try a boosting-based classifier.
Can you write Python code for it?


I am working on the file "SpIESHighzQuasarPhotoz.ipynb".
The first task is:
Convert from maggies to magnitudes, use the "clean" fluxes when available, but only if >0 to avoid problems with logs.
Can you write Python code for it?


I am working on the file "SpIESHighzQuasarPhotoz.ipynb".
The first task is:
Cut some questionable data
Can you write Python code for it?

mask2 = ((data3['u']>35) | (data3['g']>30) | (data3['r']>30) | (data3['i']>30) | (data3['z']>30))
data3 = data3[mask2]

X = np.vstack([ data3['u'], data3['g'], data3['r'], data3['i'], data3['z']]).T
y = np.array(data3['ZSPEC'])

ug = np.array(data3['u']-data3['g'])
gr = data3['g']-data3['r']

fig = plt.figure(figsize=(6,6))
#plt.scatter(y,(X[:,0]-X[:,1]))
plt.scatter(y,ug)
plt.xlabel('Redshift')
plt.ylabel('u-g')
plt.show()

from sklearn.model_selection import train_test_split, cross_val_predict
from sklearn.metrics import classification_report

# Split the training data into training and test sets for cross-validation
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.25, random_state=42)


I am working on the file "SpIESHighzQuasarPhotoz.ipynb".
The first task is:
Known Quasars II
Hmm, actually I think that it makes more sense to use the training data as the input for a photo-z algorithm.  Because I've already weeded out the bad data.  The $z<3.5$ quasars are labeled as "stars", but so what.  The quasars are all still in the file.
So, as in the SpIESHighCandidateSelection notebook, we want to use<br>
GTR-ADM-QSO-ir-testhighz_findbw_lup_2016_starclean.fits
Can you write Python code for it?

# Read in data file
%matplotlib inline
from astropy.table import Table
import numpy as np
import matplotlib.pyplot as plt
data = Table.read('GTR-ADM-QSO-ir-testhighz_findbw_lup_2016_starclean.fits')

# Remove stars
qmask = (data['zspec']>0)
qdata = data[qmask]
print len(qdata)

# X is in the format need for all of the sklearn tools, it just has the colors
X = np.vstack([ qdata['ug'], qdata['gr'], qdata['ri'], qdata['iz'], qdata['zs1'], qdata['s1s2']]).T
#y = np.array(data['labels'])
y = np.array(qdata['zspec'])

from sklearn.model_selection import train_test_split, cross_val_predict
from sklearn.metrics import classification_report
# Split the training data into training and test sets for cross-validation
Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, test_size=0.20, random_state=42)

# For algorithms that need scaled data:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(Xtrain)  # Don't cheat - fit only on training data
XStrain = scaler.transform(Xtrain)
XStest = scaler.transform(Xtest)  # apply same transformation to test data

#Include more attributes to see if that helps at all with classification
#XX = np.vstack([ data['ug'], data['gr'], data['ri'], data['iz'], data['zs1'], data['s1s2'], data['imag'], data['extinctu'], data['morph']]).T

fig = plt.figure(figsize=(6,6))
plt.scatter(y,qdata['ug'])
plt.xlabel('Redshift')
plt.ylabel('u-g')
plt.show()

# Same plot, but with KDE

Xplot = np.vstack([ qdata['zspec'], qdata['ug'] ]).T

from sklearn.neighbors import KernelDensity
kde = KernelDensity(kernel='gaussian', bandwidth=0.1)
kde.fit(Xplot) #fit the model to the data
u = np.linspace(-0.1,7,100)
v = np.linspace(-2.5,7.5,100)
Xgrid = np.vstack(map(np.ravel, np.meshgrid(u, v))).T
dens = np.exp(kde.score_samples(Xgrid)) #evaluate the model on the grid

fig = plt.figure(figsize=(6,6))
plt.scatter(Xgrid[:,0],Xgrid[:,1], c=dens, cmap="Purples", edgecolor="None")
plt.xlabel('Redshift')
plt.ylabel('u-g')
plt.xlim([0,5])
plt.ylim([-0.5,3])
plt.colorbar()
plt.show()


I am working on the file "SpIESHighzQuasarPhotoz.ipynb".
The first task is:
Normally, I compute the fraction that are within $\Delta z$ of 0.1 or 0.3 as my diagnostic.   Ideally shooting for 90% within 0.3.
Can you write Python code for it?

n = len(ypredselfNW)
mask1 = (np.abs(ypredselfNW-ytest)<0.1)
mask3 = (np.abs(ypredselfNW-ytest)<0.3)
m1 = len(ypredselfNW[mask1])
m3 = len(ypredselfNW[mask3])

frac1 = 1.0*m1/n
frac3 = 1.0*m3/n

print frac1,frac3


I am working on the file "SpIESHighzQuasarPhotoz.ipynb".
The first task is:
Self-Evaluationa Regression Tests
Try some of the regression algorithms.  For now, using just the colors and use a sub-sample of the training data for testing.
Nadaraya-Watson
First try Nadaraya-Watson (Kernel Regression).
GTR: This uses all of my memory.  So, we'll use dask below instead.  This is just to get things set up.
Can you write Python code for it?

import numpy as np
from astroML.linear_model import NadarayaWatson
model = NadarayaWatson('gaussian', 0.05)
#model = NadarayaWatson('rbf')

model.fit(Xtrain,ytrain)

%timeit ypred = model.predict(Xtest[:500])


I am working on the file "SpIESHighzQuasarPhotoz.ipynb".
The first task is:
So 500 objects are taking 2.5s.  We need to run it on as many as 200,000 objects eventually, so that is >2500s, which is something to keep in mind!
Can you write Python code for it?

print len(Xtest)

fig = plt.figure(figsize=(6,6))
#plt.scatter(y,(X[:,0]-X[:,1]))
plt.scatter(ytest[:500],ypred)
plt.xlabel('zspec')
plt.ylabel('zphot')
plt.xlim([-0.1,5.5])
plt.ylim([-0.1,5.5])
plt.show()


I am working on the file "SpIESHighzQuasarPhotoz.ipynb".
The first task is:
Test data
We have read in the known quasars above, now we need to read in the test set objects.

If we want more than just the colors, then we can use:

Using

GTR-ADM-QSO-ir_good_test_2016n.fits

which I kludged together from

GTR-ADM-QSO-ir_classifiers_good_test_2016.dat (and .fits)

GTR-ADM-QSO-ir_others_good_test_2016.dat (and .fits)

GTR-ADM-QSO-ir_photoz_in7_good_test_2016.dat (and .fits)
This was built with a combination of
combine_test_files_STILTSa.py

combine_test_files_STILTSb.py

combine_test_files_STILTSc.py

combine_test_files_STILTSd.py

combine_test_files_STILTSn.py
Sorry Gordon of the future, I just don't feel like documenting it better than that right now.  I had a hard time making a single STILTS script to do this all at once.  Basically the photo-z file is too big and I only needed a couple of columns from it.  I had to split each of the files into pieces, then combine them back together row by row, keeping just the rows that I needed.

If we just want the colors, then use instead:
GTR-ADM-QSO-ir_good_test_2016.fits

See the previous notebooks for more discussion.

But really we just need to compute the photo-z for the objects classified as quasars that are also in Stripe 82.  That will make the process much, much faster.  We can worry about the larger data set later.

So, instead we'll use
GTR-ADM-QSO-ir_good_test_2016_out_Stripe82all.fits

and we'll limit it to objects classified as quasars by any of the 3 methods.

But for this sandbox, we don't really need any of these.  We are just going to do self-fitting of the training set data.
Can you write Python code for it?

#testdata = Table.read('GTR-ADM-QSO-ir_good_test_2016.fits')
testdata = Table.read('GTR-ADM-QSO-ir_good_test_2016_out_Stripe82all.fits')

print testdata.keys()

qsocandmask = ((testdata['ypredRFC']==0) | (testdata['ypredSVM']==0) | (testdata['ypredBAG']==0))
testdatacand = testdata[qsocandmask]
print len(testdata),len(testdatacand)

# How many of these are known quasars?
print len(testdata[testdata['knownqso']==1])
print len(testdatacand[testdatacand['knownqso']==1])


I am working on the file "SpIESHighzQuasarPhotoz.ipynb".
The first task is:
Use Coleman and Tina's densityplot instead.  Don't know why it won't do the color bar or why it won't show the full plot at once.
Can you write Python code for it?

from densityplot import *
from pylab import *

fig = plt.figure(figsize=(6,6))
hex_scatter(y,qdata['ug'], min_cnt=10, levels=2, std=True, smoothing=1,
            hkwargs={'gridsize': 100, 'cmap': plt.cm.Blues},
            skwargs={'color': 'k'})
plt.colorbar()


I am working on the file "SpIESHighzQuasarPhotoz.ipynb".
The first task is:
We will use dask to handle the fact that we don't have enough memory to process the full array size at once and the amount of free disk for swap is also insufficient.
In short, we define the task that we want to run, we created dask objects with those tasks for each object and then we do the computation.
Can you write Python code for it?

from dask import compute, delayed

def process(Xin):
    return model.predict(Xin)

# Create dask objects
# Reshape is necessary because the format of x as drawm from Xtest
# is not what sklearn wants.
dobjs = [delayed(process)(x.reshape(1,-1)) for x in Xtest]
#print dobjs

#%%timeit
import dask.threaded
ypredselfNW = compute(*dobjs, get=dask.threaded.get)

# The dask output is sort of odd, so this is just to put the result back into the expected format.
ypredselfNW = np.array(ypredselfNW).reshape(1,-1)[0]

print len(ypredselfNW),ypredselfNW.max()


I am working on the file "SpIESHighzQuasarPhotoz2.ipynb".
The first task is:
Apply Photo-z Algorithm(s)
Random Forest
Can you write Python code for it?


I am working on the file "SpIESHighzQuasarPhotoz2.ipynb".
The first task is:
Applying to Quasars Candidates
Quasars candidates from the legacy KDE algorithm are in

GTR-ADM-QSO-ir-testhighz_kdephotoz_lup_2016_quasar_candidates.dat
Quasars candidates from the Random Forest Algorithm are in

GTR-ADM-QSO-ir_good_test_2016_out.fits
Quasar candidates from the RF, SVM, and/or bagging algorithms are in

GTR-ADM-QSO-ir_good_test_2016_out_Stripe82all.fits

In the case of the latter file, this includes Stripe82 only.  If we run on the other files, we might want to limit to Stripe 82 to keep the computing time reasonable.
Can you write Python code for it?

testdata = Table.read('GTR-ADM-QSO-ir_good_test_2016_out_Stripe82all.fits')

qsocandmask = ((testdata['ypredRFC']==0) | (testdata['ypredSVM']==0) | (testdata['ypredBAG']==0))
testdatacand = testdata[qsocandmask]
print len(testdata),len(testdatacand)

Xtest = np.vstack([ testdatacand['ug'], testdatacand['gr'], testdatacand['ri'], testdatacand['iz'], testdatacand['zs1'], testdatacand['s1s2']]).T
XStest = scaler.transform(Xtest)  # apply same transformation to test data


I am working on the file "SpIESHighzQuasarPhotoz2.ipynb".
The first task is:
Instantiate Photo-z Algorithm of Choice
Here using Nadaraya-Watson and Random Forests
Can you write Python code for it?

import numpy as np
from astroML.linear_model import NadarayaWatson
model = NadarayaWatson('gaussian', 0.05)

model.fit(Xtrain,ytrain)

from sklearn.ensemble import RandomForestRegressor
modelRF = RandomForestRegressor()
modelRF.fit(Xtrain,ytrain)


I am working on the file "SpIESHighzQuasarPhotoz2.ipynb".
The first task is:
Only need this if Xtest is too big
Can you write Python code for it?

from dask import compute, delayed
def process(Xin):
    return model.predict(Xin)

# Create dask objects
dobjs = [delayed(process)(x.reshape(1,-1)) for x in Xtest]

import dask.threaded
ypred = compute(*dobjs, get=dask.threaded.get)

# The dask output needs to be reformatted.
zphotNW = np.array(ypred).reshape(1,-1)[0]

testdatacand['zphotNW'] = zphotNW
testdatacand['zphotRF'] = zphotRF

testdatacand.write('GTR-ADM-QSO-ir_good_test_2016_out_Stripe82all_zphot.fits', format='fits')

from densityplot import *
from pylab import *
fig = plt.figure(figsize=(5,5))
hex_scatter(testdatacand['zphotNW'],testdatacand['ug'], min_cnt=10, levels=2, std=True, smoothing=1,
            hkwargs={'gridsize': 100, 'cmap': plt.cm.Blues},
            skwargs={'color': 'k'})
plt.xlabel('zphot')
plt.ylabel('u-g')
#plt.xlim([-0.1,5.5])
#plt.ylim([-0.1,5.5])
plt.show()

from astroML.plotting import hist as fancyhist
fancyhist(testdatacand['zphotRF'], bins="freedman", histtype="step")


I am working on the file "SpIESHighzQuasarPhotoz2.ipynb".
The first task is:
Since we are running on separate test data, we don't need to do a train_test_split here.  But we will scale the data.  Need to remember to scale the test data later!
Can you write Python code for it?

# For algorithms that need scaled data:
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(Xtrain)  # Don't cheat - fit only on training data


I am working on the file "SpIESHighzQuasars.ipynb".
The first task is:
Second, load the test data
Test Data
Test set data set was made as follows (see 18 April 2016 README entry):
maketest_2016.py

Output is:
classifiers_out = open('GTR-ADM-QSO-ir_classifiers_good_test_2016.dat','w')
others_out= open('GTR-ADM-QSO-ir_others_good_test_2016.dat','w')
czr_out = open('GTR-ADM-QSO-ir_photoz_in7_good_test_2016.dat','w')

Really need the first two files combined (so that we have both RA/Dec and colors in one place).
But couldn't merge them with TOPCAT or STILTS.  So had to break them into 3 pieces (with TOPCAT),
then used combine_test_files_STILTS.py to merge them together (just changing the input/output file names by hand).
Actually ran this on dirac so that I'd have more memory than on quasar.  Copied the output files back to quasar and merged them together with TOPCAT.
So

GTR-ADM-QSO-ir_others_good_test_2016a.dat + GTR-ADM-QSO-ir_classifiers_good_test_2016a.dat

gives

GTR-ADM-QSO-ir_good_test_2016a.dat

(and so on for "b" and "c").
Then

GTR-ADM-QSO-ir_good_test_2016a.dat + GTR-ADM-QSO-ir_good_test_2016b.dat + GTR-ADM-QSO-ir_good_test_2016c.dat

gives

GTR-ADM-QSO-ir_good_test_2016.dat
N.B.  Had to delete the intermediate files from quasar to save disk space.
GTR: Re-did this with combine_test_files_STILTSn.py (on quasar) since I wanted to include the i-band and ch2 magnitudes.  The output file is still GTR-ADM-QSO-ir_good_test_2016.dat.
Now read in the test file and convert it to an appropriate array format for sklearn.
Can you write Python code for it?

data2 = Table.read('GTR-ADM-QSO-ir_good_test_2016.fits')
#data2 = Table.read('GTR-ADM-QSO-ir_classifiers_good_test_2016.fits')
#data2 = Table.read('gtr-adm-qso-ir_photoz_in7_good_test_2016.fits')

Xtest = np.vstack([ data2['ug'], data2['gr'], data2['ri'], data2['iz'], data2['zs1'], data2['s1s2']]).T

print len(Xtest),Xtest[:5]


I am working on the file "SpIESHighzQuasars.ipynb".
The first task is:
See if we can even handle the first million objects, let alone the full data set.
Can you write Python code for it?

print dobjsBAG[:5]
dobjsBAG1 = dobjsBAG[:1000000]

import dask.threaded
ypredBAG1 = compute(*dobjsBAG1, get=dask.threaded.get)

ypredBAG1 = np.array(ypredBAG1).reshape(1,-1)[0]

print ypredBAG1[:100]


I am working on the file "SpIESHighzQuasars.ipynb".
The first task is:
The main test set classification ends here.  But see below for some experimenting with bagging.
Also, see

SpIESHighzQuasarsS82all.py

for some stand-alone code that classifies Stripe 82 candidates by RF, SVM and bagging.

Bagging classifier
If we want to try bagging instead, here is how we'd do that.  But this takes a really long time unless we restrict ourselves to just Stripe 82.  We'll use dask to schedule the processes, otherwise it kills the kernel.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X)  # Use the full training set now
XS = scaler.transform(X)
XStest = scaler.transform(Xtest)

from sklearn.ensemble import BaggingClassifier
from sklearn.neighbors import KNeighborsClassifier
bag = BaggingClassifier(KNeighborsClassifier(n_neighbors=5), max_samples=0.5, max_features=1.0)

bag.fit(XS, y)

from dask import compute, delayed

def processBAG(Xin):
    return bag.predict(Xin)

# Create dask objects
# Reshape is necessary because the format of x as drawm from Xtest
# is not what sklearn wants.
dobjsBAG = [delayed(processBAG)(x.reshape(1,-1)) for x in XStest]
#print dobjs


I am working on the file "SpIESHighzQuasars.ipynb".
The first task is:
This gives about 27,000 candidates in all and about 3000 on Stripe 82.
You can get the output file from
http://www.physics.drexel.edu/~gtr/outgoing/spies/GTR-ADM-QSO-ir_good_test_2016_out.fits.bz2
Can you write Python code for it?

mask = (data2['ypred']==0)
quasars = data2[mask]
print len(quasars)


I am working on the file "SpIESHighzQuasars2.ipynb".
The first task is:
I had some problems with GTR-ADM-QSO-ir_good_test_2016n.fits because it thought that there were blank entries among the attributes.  There actually weren't (as far as I could tell), but I found that I could use filled to fix the problem.  However, that just caused problems later!
Can you write Python code for it?

# Not sure why I need to do this because there don't appear to be any unfilled columns
# but the code segment below won't run without it.
# Only need to do for the file with imag and extinctu
# data2 = data2.filled()


I am working on the file "SpIESHighzQuasars2.ipynb".
The first task is:
Now write results to output file.  Didn't do bagging b/c takes too long.  See SpIESHighzQuasarsS82all.py which I ran on dirac.
Can you write Python code for it?

dataS82['ypredSVM'] = ypredSVM
dataS82['ypredBAG'] = ypredBAG
#dataS82.write('GTR-ADM-QSO-ir_good_test_2016_Stripe82svm.fits', format='fits')


I am working on the file "SpIESHighzQuasars2.ipynb".
The first task is:
Quasar Candidates
Finally, do the classification and output the test file, including the predicted labels.
Can you write Python code for it?

from dask import compute, delayed

def processSVM(Xin):
    return svm.predict(Xin)

# Create dask objects
# Reshape is necessary because the format of x as drawm from Xtest
# is not what sklearn wants.
dobjsSVM = [delayed(processSVM)(x.reshape(1,-1)) for x in XStest]

import dask.threaded
ypredSVM = compute(*dobjsSVM, get=dask.threaded.get)

ypredSVM = np.array(ypredSVM).reshape(1,-1)[0]

from dask import compute, delayed

def processBAG(Xin):
    return bag.predict(Xin)

# Create dask objects
# Reshape is necessary because the format of x as drawm from Xtest
# is not what sklearn wants.
dobjsBAG = [delayed(processBAG)(x.reshape(1,-1)) for x in XStest]

import dask.threaded
ypredBAG = compute(*dobjsBAG, get=dask.threaded.get)

ypredBAG = np.array(ypredBAG).reshape(1,-1)[0]


I am working on the file "SpIESHighzQuasars2.ipynb".
The first task is:
Second, load the test data
Test Data
Test set data set was made as follows (see 18 April 2016 README entry):
maketest_2016.py

Output is:
classifiers_out = open('GTR-ADM-QSO-ir_classifiers_good_test_2016.dat','w')
others_out= open('GTR-ADM-QSO-ir_others_good_test_2016.dat','w')
czr_out = open('GTR-ADM-QSO-ir_photoz_in7_good_test_2016.dat','w')

Really need the first two files combined (so that we have both RA/Dec and colors in one place).
But couldn't merge them with TOPCAT or STILTS.  So had to break them into 3 pieces (with TOPCAT),
then used combine_test_files_STILTS.py to merge them together (just changing the input/output file names by hand).
Actually ran this on dirac so that I'd have more memory than on quasar.  Copied the output files back to quasar and merged them together with TOPCAT.
So

GTR-ADM-QSO-ir_others_good_test_2016a.dat + GTR-ADM-QSO-ir_classifiers_good_test_2016a.dat

gives

GTR-ADM-QSO-ir_good_test_2016a.dat

(and so on for "b" and "c").
Then

GTR-ADM-QSO-ir_good_test_2016a.dat + GTR-ADM-QSO-ir_good_test_2016b.dat + GTR-ADM-QSO-ir_good_test_2016c.dat

gives

GTR-ADM-QSO-ir_good_test_2016.dat

and similarly for the fits output file.

Since I wanted to use the imag and extinctu, then I also had to make a version of the test file with combine_test_files_STILTSn.py (on quasar).  This was fairly involved because of memory issues.  The new output file is GTR-ADM-QSO-ir_good_test_2016n.dat.  In the end, I ended up not using that and this is more of an exploration of SVM and bagging as alternatives to RF.
Now read in the test file and convert it to an appropriate array format for sklearn.
Can you write Python code for it?

#data2 = Table.read('GTR-ADM-QSO-ir_good_test_2016n.fits')
data2 = Table.read('GTR-ADM-QSO-ir_good_test_2016.fits')

print data2.keys()


I am working on the file "Space Analysis .ipynb".
The first task is:
2. Aggregate the data by hour and count the shelves for each hour
The following sub-tasks are required for this part of our analysis

Create a new column in the dataframe that contains a distinct value for each hour in the database
Group the individual shelves by hour label
Count the shelves for each hour
Plot the number of shelves per hour for the analysis period
Can you write Python code for it?

# create a dataframe that has a subset of rows that only contain valid datetime values
clean_dates = shelf_data[shelf_data.datetime.notnull()]
clean_dates = clean_dates.set_index('datetime')
#clean_dates['hour'] = clean_dates['datetime'].strftime('%Y-%M-%d %H')

clean_dates

# resample the timestamps by hour and return the frequency distribution
r = clean_dates.resample("1H")

r_count = r.count() # .loc[(r_count.timestamp > 0)]

r_count.timestamp

r_count['timestamp'].plot()


I am working on the file "Space Analysis .ipynb".
The first task is:
3. Aggregate the occupied space for all of the shelves for each range
The following sub-tasks are required for this part of our analysis:

Total the occupied space for all of the whole shelves - i.e. those where the start and end range alpha values are the same
Estimate the occupied space for all of the partial shelves
Add these numbers up  for an estimate of the total occupied space for each range
Can you write Python code for it?

full_shelves = shelf_data.loc[(shelf_data.leading_alpha == shelf_data.ending_alpha)]
partial_shelves = shelf_data.loc[(shelf_data.leading_alpha != shelf_data.ending_alpha)].loc[(shelf_data.leading_alpha.notnull())]

# convert our column to numeric values instead of the string type it defaulted to - adding strings is not
# what we want to do. Partial shelves are arbitrarily assigned half of the occupied value and the capacity.
pd.to_numeric(full_shelves['occupied_in'], errors="coerce")
partial_shelves['partial_occupied_in'] = pd.to_numeric(partial_shelves['occupied_in'], errors="coerce") / 2
partial_shelves['partial_capacity'] = pd.to_numeric(partial_shelves['capacity'], errors="coerce") / 2

#print(full_shelves[['leading_alpha','ending_alpha','occupied_in','capacity']])
#print(partial_shelves[['leading_alpha','ending_alpha','partial_occupied_in','partial_capacity']])


I am working on the file "Space Analysis .ipynb".
The first task is:
5. Repeat the above range-level analyses for the full top-level LC subject category prefixes
Can you write Python code for it?

prefix_full_shelves = shelf_data.loc[(shelf_data.leading_prefix == shelf_data.ending_prefix)]
prefix_partial_shelves = shelf_data.loc[(shelf_data.leading_prefix != shelf_data.ending_prefix)].loc[(shelf_data.leading_prefix.notnull())]

# convert our column to numeric values instead of the string type it defaulted to - adding strings is not
# what we want to do. Partial shelves are arbitrarily assigned half of the occupied value and the capacity.
pd.to_numeric(prefix_full_shelves['occupied_in'], errors="coerce")
prefix_partial_shelves['partial_occupied_in'] = pd.to_numeric(prefix_partial_shelves['occupied_in'], errors="coerce") / 2
prefix_partial_shelves['partial_capacity'] = pd.to_numeric(prefix_partial_shelves['capacity'], errors="coerce") / 2

#print(full_shelves[['leading_prefix','ending_prefix','occupied_in','capacity']])
#print(partial_shelves[['leading_prefix','ending_prefix','partial_occupied_in','partial_capacity']])

# import a useful library for doing calculations on arrays - including our data table and its columns
import numpy as np

# group our full and partial shelves by the leading_alpha subject string
prefix_full_shelves_by_range = prefix_full_shelves.groupby('leading_prefix')
prefix_partial_shelves_by_range = prefix_partial_shelves.groupby('leading_prefix')

# calculate the sum for all of the numeric columns for our full and partial shelves
prefix_full_ranges = prefix_full_shelves_by_range.aggregate(np.sum)
prefix_partial_ranges = prefix_partial_shelves_by_range.aggregate(np.sum)

# calculate some new columns of data for each range
prefix_combined_by_lc = pd.concat([prefix_full_ranges,prefix_partial_ranges], axis=1)
prefix_combined_by_lc['total_occupied_in'] = prefix_combined_by_lc[['occupied_in','partial_occupied_in']].sum(axis=1)
prefix_combined_by_lc['total_capacity_in'] = prefix_combined_by_lc[['capacity','partial_capacity']].sum(axis=1)
prefix_combined_by_lc['total_empty_in'] = prefix_combined_by_lc['total_capacity_in'] -...


I am working on the file "Spacy.ipynb".
The first task is:
Display dependency graph
Can you write Python code for it?

displacy.render(nlp(str(sentence)), style='dep', jupyter = True, options = {'distance': 120})

[(x.orth_,x.pos_, x.lemma_)
     for x in [y for y in nlp(sentence)
               if not y.is_stop and y.pos_ != 'PUNCT']]