I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Exercise
Declare and instantiate variables to store a users details (name, height and year of birth)
- Print the name
- Print their height if they are under 7ft or print Wow, just wow if they are 7ft or over
- Calculate their (approximate, you do not have the month) age from the year of birth
- Print their age if they are 30 or over or print YOUNGUN if they are under 30
Can you write Python code for it?

name = "Jill"
height = 8
yob = 1990
year = 2016
print(name)
if height < 7:
    print(height)
else:
    print("WOW, JUST WOW")
age = year - yob
if age >= 30:
    print(age)
else:
    print("YOUNGUN")


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
If the user is 18 or over then we print that they can come in. What about the users under 18? They get no output. How do we get around this?
Else
The else statement. This is the catch all statement for when the statement before is not true.
You should always have one!
Can you write Python code for it?

age = 12
if age >= 18:
    print("come on in")
else:
    print("I think this ID is fake")


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
If you want to know more about comparing strings like this, then look at this
Equality Operators
So, how do we know when something is not equal?
Can you write Python code for it?


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Is and Is not
== and != are not the only ways to check for equality.
Can you write Python code for it?

print(2 is 2)
name = "Chris"
print(name is not name)


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Precedence
When you see if statements in most other languages you are probably used to seeing this:
if(x < 3)
Python does care about brackets but not in the same way.
if(x > 10):
is the same as
if x > 10:
Then what are brackets for?
Precedence, or order of evalutation, is exactly the same as BODMAS (Brackets, Order, Division, Multiplication, Addition and Subtraction) from GCSE Maths.
Can you write Python code for it?

print(5 + 6 * 9)
print((5 + 6) * 9) #see how the brackets change the order? This applies to if statements as well


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Take note of the structure
- COLON : at the end of the if statement
- indent (4 spaces) for code you want to run only if the statement is true
- unindent to go back to the normal running
Exercise
What are the benefits of if statements?
What kind of control is this?
Write an if statement that prints hello and the name  if the name variable is equal to Winston Churchill
Flow Control
Python is interpreted.
What we have written so far runs your code line by line. This is known as the flow of the program
If statements will only run the block of code when the statement is true
This interrupts the flow of the program
Can you write Python code for it?

name = "Homer"
age = 80

print("My name is " + name)
print("I am %d" % age)
print("I am over one hundred years old!") #is this true? With no flow control it will print anyway

name = "Homer"
age = 80

print("My name is " + name)
print("I am " + age)
if age > 100:
    print("I am over one hundred years old!") #is this true? With no flow control it will print anyway
    print("this will print if Homer is over 100, too") #as long as you stay indented, this is all part of the if
print("this will print regardless")


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
These are both comparing the same thing? Why is one True and the other is False?
Equality comparison is only concerned with the value
Identity comparison is only concerned with the object. If 2 objects have the same value but are not stored in the same memory address then they are not equal. This can be explained further by...
Immutable vs Mutable
What does this mean?
Immutable means that something cannot be changed after it is created
Mutable means that something can be changed after it is created
Exercise
Give me 2 examples of immutable things
Give me 2 examples of mutable things
Immutable and Mutable in Python
Strings are immutable in Python (and other languages).
Mutable types are beneficial because you can make changes in place.
WARNING: if you reference that mutable variable from another variable and make a change then ALL references change. We will come back to this
Can you write Python code for it?

string_var = "fdjklhd" #immutable
print(string_var) #fdjklhd
s = string_var
s += "ueoureo"
print(string_var) #fdjklhd
print(s) #fdjklhdueoureo

list_var = [] # mutable and we are still learning about these later
print(list_var) #[]
l = list_var # l is a reference to list_var
l.append(1)
print(list_var) #[1]
print(l) #[1]


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
What? I hear you ask, how is this any different to ==?
Hold your horses and let's find out.
Identity vs Equality
== is equality comparison (checking if two values are the same)
is is identity comparison (checking if 2 objects are the same)
Can you write Python code for it?

a = "chris"
b = "chris"
print(a is b) # identity (True)
print(a == b) # equality (True)

a = [] # we will learn about these soon!
b = []
print(a is b) # identity (False)
print(a == b) # equality (True)


I am working on the file "Session_3/3_Homework.ipynb".
The first task is:
a.) num_wheels = 2456
b.) num_wheels = 1765
c.) num_wheels = 1024
d.) num_wheels = 8000
e.) num_wheels = 1028
f.) num_wheels = 1020
Fix the if statements below so that the checks are correct and the right statements are printed for the age and drivers info.
You may also want to look at the precedence section in the lecture to ensure it is right in all situations.
You DO NOT need to edit variables, just if statements.
Can you write Python code for it?

age = 23
has_license = True
min_drive_age = 17
cheap_ins_age = 25

def age_check(age):
    if age >= 18:
        print("Yay, you can drink legally!")
        return True
    else:
        return False
def drive_check(age, has_license):
    if age < min_drive_age and has_license:
        print("Not legally able to drive but have a license? Do you think I am a fool?")
        return False
    if age >= min_drive_age and not has_license:
        print("You can drive but you need to get your license!")
        return False
    elif age < cheap_ins_age and has_license and age >= min_drive_age: #precedence can be used here to make things cleaner
        print("You can drive, but insurance is not gonna be cheap")
        return True
    elif age >= cheap_ins_age and has_license:
        print("Go you, cheap insurance all round!")
        return True
    else:
        return False


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Accessing Elements in a List
How would we print the last element of the list in the previous example?
print(movie_list[4])
What if we do not know the size of the list?
Can you write Python code for it?

# Python knows the size

print(movie_list[len(movie_list)])

print(len(movie_list))

# Why will this not work?


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Accessing Elements in a List
Reverse Indexing
Using the method from the previous slides is pretty longwinded, although useful. This is the way to do it in most programming languages.
However, Python is not always like most programming languages. In this case, we have reverse indexing! Which works just how it sounds.
Can you write Python code for it?

print(movie_list[-1]) #last element
print(movie_list[-2]) #second to last


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Adding to Lists
Can you write Python code for it?

awful_movies = []
awful_movies.append('The Artist')
awful_movies.append('The Room')
awful_movies.append('Rambo 4')
print(awful_movies)


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Controlling Iteration
Sometimes, we may only want to execute when certain elements match some criteria we define. Like in the example above, we can use if statements for this, but there are some other keywords
break
Using the break keyword, we can break out of our loop before it has finished. Why might this be useful?
Can you write Python code for it?

bad_words = ['fudge', 'bar steward', 'doody head']
my_string = input()
my_string_list = my_string.split(' ') #split at space to create a list of words
for word in my_string_list:
    if word in bad_words:
        print("Whoa now, no need for obscenities")
        break
    else:
        print(word)


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Exercise
Create the list above. Using the dir and help functions, do the following:
- Copy the list to a new list and call it great_movies
- Pop the last element off the new list
- What is the method to count the number of occurrences of an item in a list?
- Print the number of occurrences of Rambo 4
Can you write Python code for it?

great_movies = awful_movies.copy()
print(my_film.pop())
print(awful_movies)
#count('item_to_find')
print(awful_movies.index('Rambo 4'))


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Exercise
What index is the Godfather?
Print the 4th element
Make the second element equal Toy Story
Print the length of the list
Can you write Python code for it?

# 0
print(movie_list[3])
movie_list[1] = "Toy Story"
print(len(movie_list))


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Exercise
Write a for loop that asks a user for 5 positive integers and adds them to a list
Break out of the loop if the user input is not an integer
Print the mean if all 5 are input
Hint:
- Write the code to add to the list first
- You can check if a string is a positive integer using isdigit
What other ways could we use to check if the input variable is a number?
Can you write Python code for it?

user_nums = []
total = 0
for i in range(0, 5):
    user_input = input("Enter a positive integer please: ")
    if user_input.isdigit():
        user_nums.append(user_input)
        total += int(user_input)
    else:
        print("I said POSITIVE NUMBER!")
        break
if len(user_nums) == 5:
    print(total/len(user_nums))


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Exercise
Write a for loop that only prints out only the consonants of a word input by the user.
Hint: looping through a string using a for...in loop gives you each letter
Can you write Python code for it?

user_string = input()
vowels = ['a', 'e', 'i', 'o', 'u']
for letter in user_string:
    if letter in vowels:
        continue
    print(letter)


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
HINT: CTRL-C ends a script
Sometimes, infinite loops can be useful. For example, we may want to keep receiving information from a client if we are listening for new files or input from a user if we are making a text-based game
Exercise
Write a while loop that prints what a user has typed.
If the user types quit then exit the loop
If the user enters dnp then do not print but still prompt them to enter more text
Can you write Python code for it?

while(True):
    user_string = input("Tell me a secret: ")
    if user_string == "quit":
        break
    elif "dnp" in user_string:
        continue
    else:
        print(user_string)


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Infinite Loops
Be careful with while loops as they do not change the value like a for loop does!
Can you write Python code for it?

i = 0
while(i < 10):
    print(i) #how many times will i be printed?


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Iterating Over Lists
As expected, there is no silver bullet, each of these solutions is perfectly viable. However, when needing both index and value, enumerate is recommended because it supports multiple data types.
There are many studies that show the memory usage and execution time of range and enumerate, but that is beyond the scope of this course.
Exercise

Create a list of some cities in the UK (use people at your table)
Loop through them and print them out
Print the index of the city if it contains the letter h
Can you write Python code for it?

cities = ['Manchester', 'Cardiff', 'London', 'Southampton']
for key,value in enumerate(cities):
    print(value)
    if 'h' in value:
        print("H found at index: {0}".format(key))


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
List Comprehension
Lists in Python are more powerful than general arrays in most other languages. Lists allow you to perform comprehensions, which are really just instructions that generally cover multiple lines, condensed down into a single line. For example, consider creating a list of all odd numbers between 1 and 100.
How might we do this?
Can you write Python code for it?

odd_list = []
# for i in range(1, 100, 2):
#    odd_list.append(i)

for i in range(1, 100):
    if i % 2 != 0:
        odd_list.append(i)

print(odd_list)

odd_list = [x for x in range(1, 100, 2)] # list comprehension is comprehensive!
print(odd_list)


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Lists and Loops
Dr. Chris Gwilliams
gwilliamsc@cardiff.ac.uk
Overview So Far

Introduction to Python
Types
Variables
Functions (built in and your own)
Methods
Scope
Imports

This Session

Lists
indexing
negative indexing
Operations and length

Loops
Iteration
Recursion
Understanding the difference

Input
First thing to note: If you want the user to be able to input their own information, you can use the input function to store their result.
python
name = input("Please add your name: ")
print(name)
Exercise
Create a variable that holds a value. Ask the user for input. If the input is > 20, multiple the two and print the result. Otherwise, divide the two and print it.
Storing Multiple Values
Exercise
Write a script that stores the names of 5 of your peers.
python
name_one = "Sarah"
name_two = "Rick"
name_three = "Morty"
- Now how would you do that for 100?
Lists
Lists (also known as arrays) are data structures that can hold a collection of literals (often called elements).
my_list = ['1', 2,  True]
my_list = []
Can be instantiated with values or declared empty.
Zero-indexed and accessed with square brackets ([])
mylist[0] #'1'
Exercise
Lists are mutable, but strings are immutable.
What does that mean?
python
list_one = ['cat', 'dog', 'hamster']
list_two = list_one
list_two.append('narwhal')
print(list_one)
What will the code above output?
Lists are mutable, which means you can change their content without changing how they are identified.
Immutable types, like strings, cannot be changed without the identity of the variable also changing.
The code would print ['cat', 'dog', 'hamster', 'narwhal'], despite it being a different variable, all the references point to the same place!
Accessing Elements in a List
Square bracket notation
We use square brackets and the number of the index (or key) in the list in order to access the value.
Can you write Python code for it?

movie_list = ['The Godfather', 'Jaws', 'Troy', 'Midnight Special', 'Casper']


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
Lists in Python are zero-indexed. But if the len function told us there were 4 elements (including 0), that would be confusing!
So, the length of a list may be 5 elements but we have to access them using zero-indexed indices. So, how would we get the last element?
Can you write Python code for it?


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
The best way to think of negative indexing is to see:
list[-X]
as
list[len(list)-X]
Exercise

Instantiate a list with 3 elements inside
Print the first element
Change the first element to equal house
Print the whole list
Print the length of the list
Print the second to last element
Create an empty list
What happens when you try and print the second element?
What about when you assign a literal to the first element?
Can you write Python code for it?

three_elements = [1, 2, 3]
print(three_elements[0])
three_elements[0] = 'house'
print(three_elements)
empty_list = []
print(empty_list[1]) #IndexError
empty_list[0] = 1 #IndexError


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
While Loops
While loops are another form of iteration, calling a block of code while a condition is true.
All of the above keywords can be used within while loops and other Python code.
Can you write Python code for it?

i = 0
while(i < 6):
    print(i)
    i += 1


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
continue
The continue keyword passes over the current iteration and heads straight for the next one. For example, if a value matches a value we do not want.
The difference here is that the loop does not stop. The loop continues until the end and just skips when continue is used.
Can you write Python code for it?

for i in range(0, 5):
    if i == 2:
        continue #head back to the top and go to the next value for i
    print(i) #if i == 2 then this line is not executed


I am working on the file "Session_4/4_Lists and Loops.ipynb".
The first task is:
pass
pass is a keyword used when we do not want to execute a block of code.
Typically, you would not use this in a loop but it can have benefits when you are developing.
For example: what happens if you create a function but do not write any code for it yet?
Can you write Python code for it?

def my_function():

def my_function():
    pass


I am working on the file "Session_5/5_Homework.ipynb".
The first task is:
What would change if you added the line global name into the function update_name?
Write a script below that asks a user for 2 numbers, these will not be integers by default. Pass those numbers to a function, called power, that returns the first argument to the power of the second. You will need to look up the symbol for to the power of in the Python docs
Can you write Python code for it?

num_one = None
num_two = None

def power(one, two):
    pass

from nose.tools import assert_equal, assert_not_equal

assert_equal(square(12,2), 144)
assert_equal(square(50, 4), 6250000)
assert_equal(square(2,2), 4)


I am working on the file "Session_5/5_IO, Formatting Strings and Functions.ipynb".
The first task is:
Casting Types
Using built in functions, we can convert values from one type to another.
Can you write Python code for it?

print(str(1)) #convert int to string
print(int(True)) #bool to int
print(int("12")) #string to int


I am working on the file "Session_5/5_IO, Formatting Strings and Functions.ipynb".
The first task is:
Checking types
We can use the built in type function to check what type a value is:
Can you write Python code for it?

print(type(1))
print(type("what am I?"))
print(type(True))


I am working on the file "Session_5/5_IO, Formatting Strings and Functions.ipynb".
The first task is:
Methods
Methods are functions that are attached to objects. While we call a function like this:
function(argument1, argument2)
Methods are called like this:
object.method(argument1, argument2)
Objects in Python have methods attached to them that you can use. Let's learn about methods attached to strings!
What is the name of the function in Python that lists the methods attached to an object?
String Formatting
Creating a string like this:
python
bit_to_add = "something else"
my_string = "something something " + bit_to_add + " blah blah blah"
This is the more basic way of handling strings and it means we have to deal with different types (that we will cover later in the session)
.format
Python now has support to do something commonly known as string formatting.
Format is a method (we will learn about these as a concept later) built into the string type that allows you to specify what it contains.
Can you write Python code for it?

print("My name is {}".format('Fred'))
print("My name is {} {}".format(4, 'Terry'))
print("My name is {0}".format(True)) # format does not care about types


I am working on the file "Session_5/5_IO, Formatting Strings and Functions.ipynb".
The first task is:
Scope
This is a practical example of local scope vesus global scope. age is a global variable that has the value: 200.
Within the change_age function, a new variable, also called age, is created that is only within the local scope.
How do we access the global variable  within a function?
Can you write Python code for it?

age = 200

def change_age():
    global age
    age = "test"

print(age)
change_age()
print(age)

def change_age_no_glob(age):
    age = "test"
    return age

age = change_age_no_glob(age)


I am working on the file "Session_5/5_IO, Formatting Strings and Functions.ipynb".
The first task is:
Scope
local_var was created within the scope of a_function and was not in the global scope, so it could not be printed.
Exercise

Write a function that delcares a variable within it
Return the variable from the function and assign it to a global variable
Print the global variable

Now you have changed the scope of a variable from local to global
Can you write Python code for it?

def func():
    local = "I am local"
    return(local)
global_var = func()
print(global_var)

age = 200

def change_age():
    age = "test"
print(age)
change_age()
print(age) #What should this equal?


I am working on the file "Session_5/5_IO, Formatting Strings and Functions.ipynb".
The first task is:
function
method
method
function
method
function

Casting Types
We can instantiate a variable to be of a particular type, and we can change the type by changing the value assigned, like so:
Can you write Python code for it?

pet_name = "Mr. Woofington"
print("Hello " + pet_name)
pet_name = 14055
print("Hello " + pet_name) #Why is this?


I am working on the file "Session_6/Modules, Imports and Packages.ipynb".
The first task is:
Exercise
Using the dir method (and the documentation), import the random module and generate a random number between 42 and 749.
Can you write Python code for it?

import random #import section

print(random.randrange(42,749)) #code section

from random import randrange

print(randrange(10,300))


I am working on the file "Session_6/Modules, Imports and Packages.ipynb".
The first task is:
Installing packages
Ever used aptitude or yum on Linux? These are package managers that allow you to extend the functionality of the system you are using. Python has these, in the form of pip and easy_install.
Exercise
Which one of these should you use? (Cite your sources)
| pip                                | easy_install                                                               |
|----------------------------------------|--------------------------------------------------------------------------------|
| actively maintained                    | partially maintained                                                           |
| part of core python distribution       | support for version control                                                    |
| packages downloaded and then installed | packages downloaded and installed asynchronously                               |
| allows uninstall                       | does not provide uninstall functionality                                       |
| automated installing of requirements   | if an install fails, it may not fail cleanly and leave your environment broken |
pip
The recommended tool for installing Python packages.
Stands for Pip Installs Packages. Packages can be found on PyPi
Usage:
pip install <package-name>
Note: Some packages will require administrator privilges to be installed.
Exercise

Install a package called blessings
Find the documentation on PyPi
Write a script that uses blessings
Make the script print Sup, World in bold
List 3 commands you can run with pip

Virtual Environments (virtual env)
Picture this: You are given a project to work on in a team. You install some packages, write some code and push it to git. Your team-mates say they cannot get it to run.
Why can they not get it to run?

Modules not installed
Modules won't install
Different operating system
Different Python version

Any number of these and more.
VirtualEnv
pip installs packages globally by default. This means your Python code is always affected by the current...
Can you write Python code for it?

import sys #system package to read command line args

print(len(sys.argv))
print(sys.argv)


I am working on the file "Session_6/Modules, Imports and Packages.ipynb".
The first task is:
Modules and Packages
A Python module is just a .py file, which you can import directly.
import config (relates to config.py somewhere on your system)
A package is a collection of Python modules that you can import all of, or just import the modules you want. For example:
import random (all modules in random package)
from random import randint (importing module from packages)
Import dos and don'ts
You can (and will) import many modules in one script, PEP asks that you follow this structure:
python
import standard_library_modules
import external_library_modules
import local_modules
More info on this and other styles can be found here
You will also see that some people will import multiple modules in one line:
import os, sys, csv, math, random
Do not do this, it makes your code hard to read and modularise
However, it is good to import multiple modules from the same package in one line:
from random import randrange, randint
Writing Your Own Local Modules
Exercise
Create a file and call it city.py.
Put some variables in there that describe a city of your choice (size, population, country etc)
Now, create a file and call it main.py.
Import your city.py file and print out the city information with formatted strings.
https://gitlab.cs.cf.ac.uk/scm7cg/cm6111_python_modules/tree/master
Can you write Python code for it?

import city

print(city.name)
print("This city has {0} people".format(city.pop))


I am working on the file "Session_6/Modules, Imports and Packages.ipynb".
The first task is:
Modules, Imports and Packages
Dr. Chris Gwilliams
gwilliamsc@cardiff.ac.uk
Python Modules
We have seen that there are many things one can do using Python, but this barely touches the surface.
Python uses modules (a.ka. libraries) to extend the basic functionality and we have 3 ways of doing this:
Standard Modules
These are modules built into the Python Standard Library, similar to the built in functions (type, len) that we have been using.
The majority of them are listed here
Exercise
Follow the link in the previous slide and find the documentation for the random module.
External Modules
These are libraries, written by developers (like you), that extend the functionality of Python. They do things like:
- Web Scraping
- Network Visualisation
- Neural Networks
- Gaming
We will cover these a bit later in the course.
Local Modules
These are .py within your file system and we will look at these later on in the session.
DO NOT EVER SAVE A SCRIPT WITH THE SAME NAME AS A MODULE YOU USE
import statements
A Python script is, typically, made up of three things at the high level:

import - modules you can use within your code
Executable code - the code you have written
Comments - ignored by the interpreter
Can you write Python code for it?


I am working on the file "Session_7a/7a_Math_Binary.ipynb".
The first task is:
1024 byte = 1 kilobyte
1024 kilobytes = 1 megabyte
1024 megabytes = 1 gigabyte
1024 gigabytes = 1 terabyte
1024 terabytes = 1 petabyte
1024 petabytes = 1 exabyte
1024 exabytes = 1 zettabyte
1024 zettabytes = 1 yotabyte
So, what about these hard drive manufacturers here? They are not alone, many follow this standard!
So, when you buy a laptop and they say it has a 3000GB hard drive (or 3TB), this may not be the case!
Exercise
How fast is your internet? Speed test it now or test the 4G on your phone.
How many mega bytes per second?
MB vs Mb
1 Megabit (Mb) is NOT equal to 1 Megabyte (MB)!
1 megabit = 1,000,000bits = 1/8 of a megabyte
Exercise
If I have a 3TB hard drive, how many 650 MB video files can I store on it?
Can you write Python code for it?

movies_per_gb = 1024.0/650.0
print(movies_per_gb) #number of videos I can store in 1GB
movies_per_tb = float(movies_per_gb * 1024)
print(movies_per_tb)
print(movies_per_tb * 3) #movies I can fit on my 3TB drive

print(((1024.0/650.0) * 1024.0) * 3.0)


I am working on the file "Session_7a/7a_Math_Binary.ipynb".
The first task is:
Binary
Everything we store (movies, text, programs), all come down to binary in their storage format. Before we get to this, we first need to know how convert numbers to binary and understand each bit.
What is the denary equivalent of this number?
0  0  0 1 0 1
Can you write Python code for it?


I am working on the file "Session_7a/7a_Math_Binary.ipynb".
The first task is:
Denary to Binary and Binary to Denary
Using the method in the slide above, we can convert by doing the following things:
1. Check the number of bits. 4 bit binary has 4 digits (0000), so we have a maximum of 15 numbers we can represent
2. Draw a table like the one in the previous slide for the number of bits
3. Check if the number you want to convert fits in to the left most bit:
    - If it does: add a 1 and subtract the number represented by the bit from the number you want to convert
    - If it does not: add a 0
4. Move along to the next bit and repeat step 3 until you get to the right most bit
Example
Convert 27 to binary.

5 bit binary can represent numbers up to 31, so we know our binary number has 5 bits.
 16  8  4  2  1

Start at 16, does 16 go into 27?
Can you write Python code for it?

Yes.
 16  8  4  2  1
 --------------
 1
Move to next bit. NUMBER = 27 - 16 = 11
Does 8 go into 11?

Yes.
16  8  4  2  1
--------------
1  1
Move to next bit. NUMBER = 9 - 8 = 1
Does 3 go into 4?

No.
16  8  4  2  1
--------------
1   1  0
Move to next bit. NUMBER still equals 1
Does 3 go into 2?

Yes.
16  8  4  2  1
--------------
1   1  0  1
Move to next bit. NUMBER still equals 1
Does 1 go into 1?

Yes.
16  8  4  2  1
--------------
1   1  0  1  1
Move to next bit. NUMBER = 1 - 1 = 0.
We are at the right most bit.

So, the binary representation for 27 is:
11011


I am working on the file "Session_7a/7a_Math_Binary.ipynb".
The first task is:
Exercise
Find the 8 bit binary representation:
* 127
* 2
* 64
* 79
* 34
Reverse engineer the approach to find the decimal representation of:

11101111
00011110
11110001
10101010
01010101
Can you write Python code for it?


I am working on the file "Session_8/8_Recursion and Dicts.ipynb".
The first task is:
Exercise
Now modify the function to use an end index as an argument (which is the length of the list to begin with).
In your check for whether the start index is more than the length of the list, do the following things:
- call the function again, with the same list as the arguments
    - the start index set to 0
    - the end index decremented
- Before returning the original list, call the function again but increment the start index
- Add a check to return the list at the start of the function, if the start index is more than the end
Can you write Python code for it?

#function receives list, start point and endpoint as args
def recursive_sort(input_list, index, end):
    #if the startpoint goes beyond the endpoint then return
    if index > end:
        return(input_list)
    #if the start point is equal to the end then decrement the end
    if index == end:
        recursive_sort(input_list, 0, end - 1)
    # check if the string at index is longer than the string at index + 1
    # replace it if it is
    # why do we need a temporary variable?
    elif len(input_list[index]) > len(input_list[index + 1]):
        current = input_list[index]
        print("Switching \"{0}\" at {1} for \"{2}\"".format(current, index, input_list[index + 1]))
        input_list[index] = input_list[index + 1]
        input_list[index + 1] = current
    # call the  function again and increment the index
    recursive_sort(input_list, index + 1, end)
    # Why do we need this here?
    return input_list

strings = ['hey', 'a', 'you there', 'what to the what to the what', 'sup', 'oi oi', 'how are you doing, good sir?']
sorted_list = recursive_sort(strings, 0, len(strings)-1)
print(sorted_list)

#uncommented
def recursive_sort(input_list, index, end):
    if index > end:
        return(input_list)
    if index == end:
        recursive_sort(input_list, 0, end - 1)
    elif len(input_list[index]) > len(input_list[index + 1]):
        current = input_list[index]
        print("Switching \"{0}\" at {1} for \"{2}\"".format(current, index, input_list[index + 1]))
        input_list[index] = input_list[index + 1]
        input_list[index + 1] = current
    recursive_sort(input_list, index + 1, end)
    return input_list

strings = ['hey', 'a', 'you there', 'what to the what to the what', 'sup', 'oi oi', 'how are you doing, good sir?']
sorted_list = recursive_sort(strings, 0, len(strings)-1)
print(sorted_list)


I am working on the file "Session_8/8_Recursion and Dicts.ipynb".
The first task is:
Exercise
Using the methods you found, write a function that has a dictionary as an argument and loops through the values to return the first value that is of type int
Create a new functiom that does the same but returns the key of that value.
Can you write Python code for it?

def find_first_int_value(dictionary):
    for val in dictionary.values():
        if type(val) is int:
            return val

def find_first_int_key(dictionary):
    for key, val in dictionary.items():
        if type(key) is int:
            return key

def example(dictionary):
    for key, val in enumerate(dictionary):
        if type(key) is int:
            return key

find_first_int_value(soft_acc)
find_first_int_key(soft_acc)
example(dictionary)


I am working on the file "Session_8/8_Recursion and Dicts.ipynb".
The first task is:
Exercise
Write a script that:
1. Users can input messages until they type 'q'
2. Messages are added to a dictionary with the length of the message as the key
3. Write a function that uses the keys as the input to return the average length
4. Write a second function that takes the values of the dictionary and sorts them according to length
Can you write Python code for it?

strings = ['hey', 'a', 'you there', 'what to the what to the what', 'sup', 'oi oi', 'how are you doing, good sir?']

# strings_dict = {}
strings_dict = {33: 'dfhjkdshfjkhdsfjkhdskahfksahkjdfk', 18: 'dkjhskjhskdhffkdjh', 19: 'dfkjsdhfkjdhsfkjhdk', 5: 'kdkdf', 6: 'fdjhfd', 9: 'fkljrwlgj', 28: 'fdjfkdjfkljlskdfjlksdjflsk;a'}
# while True:
#     msg = input("Please type your message here:")
#     if msg is not 'q':
#         strings_dict[len(msg)] = msg
#     else:
#         break

def list_to_dict(strings):
    strings_dict = {}
    for string in strings:
        strings_dict[len(string)] = string
    return strings_dict

def sort_list(input_list):
    is_sorted = True
    for key in range(0, len(input_list)):
        for i in range(0, len(input_list)):
            current = input_list[i]
            if i + 1 < len(input_list):
                if len(current) > len(input_list[i + 1]):
                    input_list[i] = input_list[i + 1]
                    input_list[i + 1] = current
    return input_list

def mean_length(lengths):
    total_length = 0
    for length in lengths:
        total_length += length
    return total_length/len(lengths)

# strings_dict = list_to_dict(strings)
print("Average string length is {0}".format(mean_length(strings_dict.keys())))
sorted_list = sort_list(list(strings_dict.values()))
print("Sorted list is {0}".format(sorted_list))


I am working on the file "Session_8/8_Recursion and Dicts.ipynb".
The first task is:
Recursion and Dictionaries
Dr. Chris Gwilliams
gwilliamsc@cardiff.ac.uk
Overview

Scripts in Python
Types
Methods and Functions
Flow control
Lists
Iteration
for loops
while loops

Now

Dicts
Tuples
Iteration vs Recursion
Recursion

Dictionaries
Python has many different data structures available (see here)
The dictionary structure is similar to a list, but the index is specified by you.
It is also known as an associative array, where values are mapped to a key.
Can you write Python code for it?

empty_dict = {}
contact_dict = {
    "name": "Homer",
    "email": "homer@simpsons.com",
    "phone": 999
}
print(contact_dict)


I am working on the file "Session_8/8_Recursion and Dicts.ipynb".
The first task is:
This follows the format of JSON (JavaScript Object Notation).
Keys can be accessed the same way as lists:
Can you write Python code for it?


I am working on the file "Session_8/8_Recursion and Dicts.ipynb".
The first task is:
What is Happening Here?
Python sees our call to the function and executes it.
Recursion Level
1
Does x equal 1? No
Return x + sum_until(2)
2
Does x equal 1? No
Return x + (2 + sum_until(1))
3
Does x equal 1? Yes
Return x (6)
Exercise
Write a function that takes a list as an input, a start index and checks if the value at that index is greater than the value at the next index. If it is more: swap them. Return the list.
HINT: You must make sure that the index + 1 must be less than the length of the list.
Can you write Python code for it?

def check_value(input_list, start):
    if(start == len(input_list) - 1):
        return
    elif(input_list[start] > input_list[start+1]):
        current = input_list[start]
        input_list[start] = input_list[start + 1]
        input_list[start + 1] = current
    return input_list

l = [3,1,4]
print(check_value(l, 0))


I am working on the file "Session_8/8_Recursion and Dicts.ipynb".
The first task is:
Why the Difference?
To understand this, we need to understand a little bit about how programs are run.
Two key things are the stack and the heap
Stack
Every time a function or a method is called, it is put on the stack to be executed. Recursion uses the stack extensively because each function calls itself (until some condition is met).
See the code below:
python
def recursive():
    return recursive()
Running this will result in the recursive function being called an infinite number of times. The Python interpreter cannot handle this, so it will shut itself down and cause a stack overflow
Heap
The heap is the space for dynamic allocation of objects. The more objects created, the greater the heap. Although, this is dynamic and can grow as the application grows.
Python also takes care of this for us, by using a garbage collector. This  tracks allocations of objects and cleans them up when they are no longer used. We can force things to be cleared by using:
del my_var
However, if assigning that variable takes up 50MB, Python may not always clear 50MB when it is deallocated. Why do you think this is?

FizzBuzz Exercise
Write a for loop that goes from 1 to 100 (inclusive) and prints:
* fizz if the number is a multiple of 3
* buzz if the number is a multiple of 5
* fizzbuzz if the number is a multiple of both 3 and 5
* the value for any other case
Exercise
Now turn this into a function and modify it to not use a for loop and use recursion. I.e. calling the function until the value reaches 100.
Homework
Insertion Sort
The insertion sort is a basic algorithm to build the sorted array in a similar way to the bubble sort.
The list is sorted by looping through all the elements from the index to the end, moving the index along for each loop.
Can you write Python code for it?

import random

def sort_numbers(s):
    for i in range(1, len(s)):
        val = s[i]
        j = i - 1
        while (j >= 0) and (s[j] > val):
            s[j+1] = s[j]
            j = j - 1
        s[j+1] = val

# x = eval(input("Enter numbers to be sorted: "))
# x = list(range(0, 10)) #list(x)
x = random.sample(range(1, 1000000001), 100000000)
print(x)
sort_numbers(x)
print(x)


I am working on the file "Session_9/9_Sorting Algorithms and Complexity.ipynb".
The first task is:
Mergesort
Merge sort is a sorting algorithm that works by splitting an array into subarrays, sorting the individual sub arrays and joining it back up into a single, sorted array.
This is known as a divide and conquer algorithm.
Can you write Python code for it?

C = []
A = [1, 3]
B = [2, 4]
C.append(min(min(A), min(B)))
print(C)

A = [3]
B = [2, 4]
C.append(min(min(A), min(B)))
print(C)

A = [3]
B = [4]
C.append(min(min(A), min(B)))
print(C)

# nothing left in A, so just add what is left of B
C += B
print(C)

def merge_sort(x):
    result = []
    if len(x) < 2:
        return x
    mid = int(len(x)/2) # calculate mid point
    y = merge_sort(x[:mid]) #split up to mid point and sort
    z = merge_sort(x[mid:]) #get the rest of the mid point and sort
    i = 0
    j = 0
    while i < len(y) and j < len(z): # loop through both lists
            if y[i] > z[j]: #if left is more than right, add to result and increment index
                result.append(z[j])
                j += 1
            else: #if right is more than left, add to result and increment right index
                result.append(y[i])
                i += 1
    result += y[i:] #we can assume this is sorted, so we add the rest of the lists based on the index
    result += z[j:]
    return result

print(merge_sort([5,8,1,3,6,9]))


I am working on the file "Session_9/9_Sorting Algorithms and Complexity.ipynb".
The first task is:
Python has sorting methods built into its standard library.
And it is pretty good!
Can you write Python code for it?

import time
import random

small_sorted_list = [x for x in range(4)]
rev_small_sorted_list = small_sorted_list[::-1]
sorted_list = [x for x in range(1024)]
rev_sorted_list = sorted_list[::-1]

#Start timing

def time_to_sort(to_sort, print_string):
    start = time.time()
    to_sort.sort()
    elapsed = time.time() - start
    print(print_string.format(elapsed))

time_to_sort(small_sorted_list, "Sorted: {}")
time_to_sort(rev_small_sorted_list, "Reverse: {}")
time_to_sort([1,5,2,8,3], "Partially Sorted: {}")
time_to_sort([random.randint(0,1000) for r in range(1024)], "Random: {}")


I am working on the file "Session_9/9_Sorting Algorithms and Complexity.ipynb".
The first task is:
Sorting Algorithms and Complexity
Dr. Chris Gwilliams
gwilliamsc@cardiff.ac.uk
Overview
We have covered:
* Iteration
* Recursion
* Functions
* Sorting Algorithms
    * Bubble Sort
    * Insertion Sort
Insertion Sort

Start at the second element and loop until the final
Compare index with value before, move it if it is greater
If j is greater than the first index, keep going until we hit the first index
Move i along and repeat 1-3

Example
Can you write Python code for it?

def ins_sort(k):
    for i in range(1,len(k)):             #since we want to swap an item with previous one, we start from 1
        j = i                             #because we need 2 indexes as one will reduce and we do not want to affect i
        while j > 0 and k[j] < k[j-1]:    #j>0 since there is no value to the left of the 0 index
            k[j], k[j-1] = k[j-1], k[j]   # swap the items, if right one is smaller. (sometimes we use a temp variable)
            j=j-1                         #reduce j and repeat
    return k

print(ins_sort([5,8,1,3,6,9]))


I am working on the file "Session_9/9_Sorting Algorithms and Complexity.ipynb".
The first task is:
sorted()
Understandably, we do not write our own sorting implementations each time we need to sort some data.
Can you write Python code for it?


I am working on the file "Sessionize-demo-2.ipynb".
The first task is:
Sessionize the table by each user_id:
Can you write Python code for it?

%%sql
DROP VIEW IF EXISTS sessionize_output_view;

 SELECT madlib.sessionize(
     'eventlog',             -- Name of input table
     'sessionize_output_view',   -- View to store sessionize results
      'user_id',             -- Partition input table by user id
     'event_timestamp',      -- Time column used to compute sessions
     '0:30:0'                -- Time out used to define a session (30 minutes)
    );

SELECT * FROM sessionize_output_view ORDER BY user_id, event_timestamp;


I am working on the file "Sessionize-demo-2.ipynb".
The first task is:
The data set describes shopper behavior on a notional web site that sells beer and wine. A beacon fires an event to a log file when the shopper visits different pages on the site: landing page, beer selection page, wine selection page, and checkout. Each user is identified by a a user id, and every time a page is visited, the page and time stamp are logged.
Create the data table:
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook I.ipynb".
The first task is:
A Toy 1-D Model
We'll make a simple one dimensional model of a star field.  Images of real stars are complicated, but we'll assume that the profile is a Gaussian.  I write a Gaussian with mean $\mu$ and variance $\sigma^2$ as $N(\mu, \sigma^2)$.
Almost all stars are essentially point sources as viewed from the Earth, so stars look like the Point Spread Function (PSF) produced by a combination of the atmosphere, the telescope, and the detector.  There's no standard notation for the PSF, but I always call it $\phi$.  The Full Width Half Maximum (FWHM) is a measure of the PSF's width.
In addition to flux from the stars that we care about there's a smooth background of extra light that comes from a number of sources (atmospheric emission, scattering of star and moonlight, dark current in the detector...); for now we'll just treat this as an annoying constant that I'll call `The Sky'.
In reality CCD data is only measured where we have pixels, but under certain conditions (band-limited; satisfying the Nyquist condition) it turns out that the pixellation is unimportant, so we'll ignore it for this session.
Task 1
Write a simulator that simulates noise-free 1-D data.  Provide a function with signature
def phi(x, xc=0.0, fwhm=2):
    """Return a numpy array with a star centred at the point xc and FWHM evaluated at the points x"""

Use a Gaussian PSF, and set the sky level to S; for a Gaussian $N(0, \beta^2)$, the FWHM is $2\sqrt{2\ln(2)}\,\beta \sim 2.3548\beta$
Plot a few of your beautiful simulations.  Once you've got it working, wrap it up in a function
simulate(x, S=100)

Answer 1
Can you write Python code for it?

def phi(x, xc=0.0, FWHM=2):
    beta = FWHM/(2*np.sqrt(2*np.log(2)))
    I = np.exp(-0.5*((x - xc)/beta)**2)
    I /= I.sum()

    return I

x = np.linspace(0, 20, 41, dtype=float)

S = 100

I = S + np.zeros_like(x)
for F, xc in [(500, 7), (2000, 15)]:
    I += F*phi(x, xc)

plt.plot(x, I,'-o')
plt.show()

def simulate(x, S=100):
    I = S + np.zeros_like(x)
    for F, xc in [(500, 7), (2000, 15)]:
        I += F*phi(x, xc)

    return I

x = np.linspace(0, 20, 41, dtype=float)
I = simulate(x)

plt.plot(x, I,'-o')
plt.show()


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook I.ipynb".
The first task is:
Answer 2
Can you write Python code for it?

x = np.linspace(0, 20, 41, dtype=float)
np.random.seed(1000)
sim = simulate(x)
I = np.random.poisson(sim)

plt.plot(x, sim)
plt.errorbar(x, I, I - sim)
plt.show()


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook I.ipynb".
The first task is:
Are those mean values significantly different from F0?  We can calculate the reduced $\chi^2$; note that we're not estimating the mean (F0) from the data, so we divide by $N$ not $N - 1$
Can you write Python code for it?

print("chi^2/nu = %.3f" % (sum(((mean/cog - F0)/(std/cog/np.sqrt(nSample)))**2)/len(RR)))


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook I.ipynb".
The first task is:
Task 3
Let's investigate measuring the flux in a single isolated star.  Start my modifying your previous answer to simulate a noisy single star with $\beta = 2$ (a FWHM of c. 5 pixels), centred at x=0, with total flux F0.  In reality estimating the sky background is not trivial, but for now let's simply subtract its known value.  Plot one of your simulations, with F0=1000 and S=100.
The simplest way to measure the flux in a star is to sum the counts within an aperture.  Modify your code to estimate the flux within 5 pixels of the centre, then run a Monte-Carlo simulation to estimate the mean and variance of your estimator.
Your mean should be close to F0 --- if it isn't, take another look at your code.  Your value won't be exactly F0; is this a statistical anomaly, or is it something more interesting?
Answer 3
Can you write Python code for it?

def simulate(x, F0 = 1000, S=100, beta=2.5):
    sim = S + np.zeros_like(x) + F0*phi(x, FWHM=2*np.sqrt(2*np.log(2))*beta)

    if True:    # I'm going to use a Gaussian approximation to the noise
        sim += np.random.normal(0, np.sqrt(S), size=len(sim))
    else:
        sim = np.random.poisson(sim + S)

    return sim - S

#np.random.seed(1000)    # uncomment to make your noise realisation repeatable

nSample = 4000
flux = np.empty(nSample)
x = np.linspace(-20, 20, 81, dtype=float)

R = 5
beta, F0 = 2.5, 1000
for i in range(nSample):
    sim = simulate(x, F0, beta=beta)
    flux[i] = np.sum(sim[np.abs(x < R)])

plt.plot(x, sim)
for r in (-R, R):
    plt.axvline(r, ls=':', color='black')
plt.show()

plt.hist(flux, 20)
plt.axvline(F0, ls=':', color='black')
plt.xlabel(r"$F_0$")
plt.ylabel("N")
mean, std = np.mean(flux), np.std(flux)
plt.title(r"R/$\beta$ = %g   %.2f $\pm$ %.2f (spread %.2f)" %
         (R/beta, mean, std/np.sqrt(len(flux)), std))
plt.show()


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook I.ipynb".
The first task is:
Task 4
Package your estimator into a function and estimate the mean and variance as a function of $R$; plot your results.  What value of $R$ gives the smallest variance in $F_0$?
For small $R$ are we measuring $F_0$ correctly?  If not, make appropriate corrections and remake your plots.  Does your conclusion change?
Can you write Python code for it?

#np.random.seed(1000)

nSample = 4000

def estimateApertureStats(x, R):
    flux = np.empty(nSample)

    for i in range(nSample):
        sim = simulate(x, F0=F0, beta=beta)
        flux[i] = np.sum(sim[np.abs(x < R)])

    return np.mean(flux), np.std(flux)

RR = np.arange(1, 16, dtype=float)
mean = np.empty_like(RR)
std = np.empty_like(RR)

x = np.linspace(-20, 20, 81)
for i, R in enumerate(RR):
    mean[i], std[i] = estimateApertureStats(x, R)

plt.errorbar(RR/beta, mean, std)
plt.axhline(F0, ls=':', color='black')
plt.xlabel(r"$R/\beta$")
plt.ylabel(r"$F_0$")
plt.show()

plt.plot(RR/beta, std)
#plt.axhline(F0, ls=':', color='black')
plt.xlabel(r"R/\beta")
plt.ylabel(r"d$F_0$")
plt.show()


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook I.ipynb".
The first task is:
Task 5
The lecture notes imply that we'd do better to use the PSF to measure the flux.  Write a Monte-Carlo simulation, and compare your results with the previous task
Answer 5
Can you write Python code for it?

nSample = 4000
flux = np.empty(nSample)
filt = phi(x, FWHM=2*np.sqrt(2*np.log(2))*beta)
filt /= np.sum(filt**2)

R = 5
beta, F0 = 2.5, 1000
for i in range(nSample):
    sim = simulate(x, F0, beta=beta)
    flux[i] = np.sum(filt*sim)

plt.plot(x, sim)
plt.plot(x, filt/np.max(filt)*np.max(sim), ls=':', color='black')
plt.show()

plt.hist(flux, 20)
plt.axvline(F0, ls=':', color='black')
plt.xlabel(r"$F_0$")
plt.ylabel("N")
plt.title(r"%.2f $\pm$ %.2f  (%.2f)" % (np.mean(flux), np.std(flux), np.sqrt(S*np.sqrt(4*pi)*beta/(x[1] - x[0]))))
plt.show()


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook I.ipynb".
The first task is:
The 0.5*(x[1] - x[0]) is to correct for the location of the sample points.
Can you write Python code for it?

cog = curveOfGrowth(RR - 0.5*(x[1] - x[0]), beta)

plt.errorbar(RR/beta, mean/cog, std/np.sqrt(nSample))
plt.axhline(F0, ls=':', color='black')
plt.ylim(F0 + 5*np.array([-1, 1]))
plt.xlabel(r"$R/\beta$")
plt.ylabel(r"$F_0$")
plt.show()

plt.plot(RR/beta, std/cog, label='Theory')
plt.plot(RR/beta, std/(mean/F0), label='Empirical')
plt.xlabel(r"R/\beta")
plt.ylabel(r"d$F_0$")
plt.legend(loc='best')
plt.show()


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook II.ipynb".
The first task is:
I lied to you; or at least I didn't tell you everything.  That 'data' was actually the output from the LSST simulator, which means that I know the Truth;  more accurately, I know the location of every photon that arrived from the sources without any sky background. The pixels are 0.2 arcseconds on a side.
Let's overlay the detection mask on the truth.
Can you write Python code for it?

t = utils.Data(image=da.truth, mask=sda.mask)
utils.mtv(t, I0=1, b=0.01, alpha=0.6)
plt.xlim(xlim); plt.ylim(ylim)
plt.show()


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook II.ipynb".
The first task is:
If you look at the direct image you can see things that seem real when you compare with the truth, for example the object at (156, 205).  So should we be using a lower threshold?  What happens if you choose a smaller value?
OK, so that picked up the object I pointed out, but it picked up some noise too.  How many false objects would I expect to detect per square degree?  Naïvely we'd expect each PSF-sized patch to be independent, so we can try using the tails of a Gaussian to estimate how many objects we'd detect per square degree.  If I take the area of a PSF to be 0.5 arcsec^2, I have
Can you write Python code for it?

import scipy.special

pixelSize = 0.200

nPerPsf = 0.5*scipy.special.erfc(nsigma/sqrt(2))
nPerDeg = nPerPsf*3600**2/0.5

print "False positives per degree: %d  In data: %d" % (
    nPerDeg, nPerDeg/(3600/(da.image.shape[0]*pixelSize))**2)


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook II.ipynb".
The first task is:
If you want to look at the raw data, you can:
Can you write Python code for it?

raw = utils.Data()
raw.read(readRaw=True)

utils.mtv(raw, b=10, alpha=0.3)
plt.xlim(740, 810); plt.ylim(230, 290)
plt.show()


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook II.ipynb".
The first task is:
Let's take a look at some data.
Rather than asking you to install a display tool such as ds9, ginga, firefly, or aladin I've provided you with a lightweight image display tool, utils.mtv()
The coloured overlay shows the mask bits that tells you which pixels are bad (its visibility is controlled
by the alpha parameter).  The stretch is controlled by "b" (it's roughly the transition from a linear to a
logarithmic stretch).
You can print the value of pixel (x, y) with
    print da.image[y, x]
(note the order of the indices).  Here x and y can be scalars or numpy arrays
Can you write Python code for it?

da = utils.Data()
da.read()

utils.mtv(da, b=10, alpha=0.8)
xlim, ylim = (80, 400), (100, 400)

plt.xlim(xlim); plt.ylim(ylim)
plt.show()


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook II.ipynb".
The first task is:
Now let's look for objects.  We know how to do this;  we smooth the image with the PSF then look for peaks.  It's not totally trivial to find all the sets of connected pixels, so I provided you with a function detection.findObjects to do the work
Can you write Python code for it?

nsigma = 3.5
threshold = nsigma*sqrt(np.median(sda.variance)/n_eff)
footprints = detection.findObjects(sda.image, threshold, grow=3)

print "I found %d objects" % (len(footprints))


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook II.ipynb".
The first task is:
Or by setting a mask plane -- this way we'll be able to see all the pixels
Can you write Python code for it?

sda.clearMaskPlane("DETECTED")
detection.setMaskFromFootprints(sda, footprints, "DETECTED")

utils.mtv(sda)

plt.xlim(xlim); plt.ylim(ylim)
plt.show()


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook II.ipynb".
The first task is:
We can also calculate the filter's effective area (and confirm or deny that I did my Gaussian integrals correctly in the lecture)
Can you write Python code for it?

phi = gaussian2D(beta)
n_eff = 1/np.sum(phi**2)
print "n_eff = %.3f (analytically: %.3f)" % (n_eff, 4*pi*beta**2)


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook II.ipynb".
The first task is:
We can look at all our objects by looping over the footprints:
Can you write Python code for it?

nShow = 10
for foot in footprints.values()[0:nShow]:
    print "(%5d, %5d) %3d" % (foot.centroid[0], foot.centroid[1], foot.npix)
if len(footprints) > nShow:
    print "..."


I am working on the file "Sessions/Session01/Day2/ImageProcessing/Image Processing Workbook II.ipynb".
The first task is:
We can show the same data without marking the bad pixels -- you'll see that I fixed them. Magic
Can you write Python code for it?

utils.mtv(da, b=10, alpha=0.0, fig=2)
plt.xlim(xlim); plt.ylim(ylim)
plt.show()


I am working on the file "Sessions/Session01/Day2/IntroStat.ipynb".
The first task is:
4.3. Any other uexpected effect
Use now the 'mag5' column in place of 'mag.outlier'. How does the standardization work in this case?
Can you write Python code for it?

w = dfr['mag5.error']**(-2)
m = np.average(dfr['mag5'], weights = w)
std5 = std_fn(x = dfr['mag5'], mean = m, std = dfr['mag5.error'])

fig = plt.figure(1)
plt.subplot(121)
plt.hist(std5, 12, normed=1, facecolor='grey', alpha=0.75)
ax = fig.add_subplot(122)
sm.qqplot(std5, line = '45', ax = ax)
fig.tight_layout()
plt.show()


I am working on the file "Sessions/Session01/Day2/IntroStat.ipynb".
The first task is:
Another useful plot (which is generally used) is the plot of residuals against covariates. We can see the intervals of lack of fits, the bias, the necessity of more terms or a nonparametric model. Create this plot. Do you see a strong indication of quadratic terms or breaks in the model?
Can you write Python code for it?

fig = plt.figure(1)

fig.add_subplot(221)
plt.plot(cep[-i_lmc & i_fu]['logP1'], cep[-i_lmc & i_fu]['resid0'], 'g*', alpha=0.2)
plt.hlines(0, xmin = cep[-i_lmc & i_fu]['logP1'].min(), xmax = cep[-i_lmc & i_fu]['logP1'].max(), lw = 2)
plt.title('SMC FU')

fig.add_subplot(222)
plt.plot(cep[i_lmc & i_fu]['logP1'], cep[i_lmc & i_fu]['resid0'], 'g*', alpha=0.2)
plt.hlines(0, xmin = cep[-i_lmc & i_fu]['logP1'].min(), xmax = cep[-i_lmc & i_fu]['logP1'].max(), lw = 2)
plt.title('LMC FU')

fig.add_subplot(223)
plt.plot(cep[-i_lmc & -i_fu]['logP1'], cep[-i_lmc & -i_fu]['resid0'], 'g*', alpha=0.2)
plt.hlines(0, xmin = cep[-i_lmc & -i_fu]['logP1'].min(), xmax = cep[-i_lmc & -i_fu]['logP1'].max(), lw = 2)
plt.title('SMC FO')

fig.add_subplot(224)
plt.plot(cep[i_lmc & -i_fu]['logP1'], cep[i_lmc & -i_fu]['resid0'], 'g*', alpha=0.2)
plt.hlines(0, xmin = cep[i_lmc & -i_fu]['logP1'].min(), xmax = cep[i_lmc & -i_fu]['logP1'].max(), lw = 2)
plt.title('LMC FO')

fig.tight_layout()

plt.show()


I am working on the file "Sessions/Session01/Day2/IntroStat.ipynb".
The first task is:
Finally, for seeing how collinearity affects the model results, we fit models also using the original V-I, correlated with log(P).
Can you write Python code for it?

lmfit_lmc_fu1 = smf.ols(formula = 'W ~ logP1 + VI', data = cep, subset = i_lmc & i_fu).fit()
lmfit_lmc_fo1 = smf.ols(formula = 'W ~ logP1 + VI', data = cep, subset = i_lmc & -i_fu).fit()

lmfit_smc_fu1 = smf.ols(formula = 'W ~ logP1 + VI', data = cep, subset = -i_lmc & i_fu).fit()
lmfit_smc_fo1 = smf.ols(formula = 'W ~ logP1 + VI', data = cep, subset = -i_lmc & -i_fu).fit()

print lmfit_lmc_fu1.summary()
print lmfit_lmc_fu2.summary()


I am working on the file "Sessions/Session01/Day2/IntroStat.ipynb".
The first task is:
Just to check on the literature, to see a frequent and sometimes unnoticed problem of linear models, and to use the model comparison techniques, we fit models with both logP1 and V - I. However, we should be careful. It cannot be excluded that the two explanatory variables directly depend on each other (actually, this can even be expected, since the Cepheids have a very constrained pulsation and stellar structure model). If such a relationship holds between two covariates in a linear model, then mathematically, the model can become ill-determined, and strongly unstable against small changes in the data. This is because in such a case, a change in the coefficient of one of these covariates can be compensated by a corresponding change in the coefficient of the other. So first use a scatterplot to see the logP1-VI relationship in the four Cepheid groups. What do you conclude?
Can you write Python code for it?

fig = plt.figure(1)

fig.add_subplot(221)
plt.plot(cep[-i_lmc & i_fu]['logP1'], cep[-i_lmc & i_fu]['VI'], 'g*', alpha=0.2)
plt.title('SMC FU')

fig.add_subplot(222)
plt.plot(cep[i_lmc & i_fu]['logP1'], cep[i_lmc & i_fu]['VI'], 'g*', alpha=0.2)
plt.title('LMC FU')

fig.add_subplot(223)
plt.plot(cep[-i_lmc & -i_fu]['logP1'], cep[-i_lmc & -i_fu]['VI'], 'g*', alpha=0.2)
plt.title('SMC FO')

fig.add_subplot(224)
plt.plot(cep[i_lmc & -i_fu]['logP1'], cep[i_lmc & -i_fu]['VI'], 'g*', alpha=0.2)
plt.title('LMC FO')

fig.tight_layout()

plt.show()


I am working on the file "Sessions/Session01/Day2/IntroStat.ipynb".
The first task is:
Next, add the new column 'resid_vi' as an additional variable to the models. Compare the different model comparison measures: the likelihood, the AIC and the BIC to those of models without resid_vi. As well, repeat the former plot, now superposing the new residuals in a new (transparent) colour. What do you see? Would you accept the necessity of including such a term into your models? Consider different aspects of the problem: improvement on the model as goodness-of-fit measures summarize, behaviour of the residuals, behaviour of the outliers, size of the effect, errors on the coefficients in the two models.
Can you write Python code for it?

cep['resid_vi'] = np.zeros(cep.shape[0])
cep.loc[(i_lmc & i_fu),'resid_vi'] = vi_lmfit_lmc_fu.resid
cep.loc[(-i_lmc & i_fu),'resid_vi'] = vi_lmfit_smc_fu.resid
cep.loc[(i_lmc & -i_fu),'resid_vi'] = vi_lmfit_lmc_fo.resid
cep.loc[(-i_lmc & -i_fu),'resid_vi'] = vi_lmfit_smc_fo.resid

cep[0:10]

lmfit_lmc_fu2 = smf.ols(formula = 'W ~ logP1 + resid_vi', data = cep, subset = i_lmc & i_fu).fit()
lmfit_lmc_fo2 = smf.ols(formula = 'W ~ logP1 + resid_vi', data = cep, subset = i_lmc & -i_fu).fit()

lmfit_smc_fu2 = smf.ols(formula = 'W ~ logP1 + resid_vi', data = cep, subset = -i_lmc & i_fu).fit()
lmfit_smc_fo2 = smf.ols(formula = 'W ~ logP1 + resid_vi', data = cep, subset = -i_lmc & -i_fu).fit()

print lmfit_lmc_fu2.summary()

print lmfit_lmc_fo.bic
print lmfit_lmc_fo2.bic


I am working on the file "Sessions/Session01/Day2/IntroStat.ipynb".
The first task is:
Replace the line by one defined using the quantile estimators (a line passing through the first and third quartiles).
Can you write Python code for it?

fig = plt.figure(1)

plt.subplot(121)
plt.hist(dfr['mag.outlier'], 12, normed=1, facecolor='y', alpha=0.75)
ax = fig.add_subplot(122)
sm.qqplot(dfr['mag.outlier'], line = 'q', ax = ax)

plt.show()


I am working on the file "Sessions/Session01/Day2/IntroStat.ipynb".
The first task is:
The above standardization should lead to a homoscedastic standard normal sample (by the linearity of the normal distribution). Its QQ-plot should be now close to a line with slope 1.
Can you write Python code for it?

w = dfr['mag.het.error']**(-2)
m = np.average(dfr['mag.het'], weights = w)
std_het = std_fn(x = dfr['mag.het'], mean = m, std = dfr['mag.het.error'])

fig = plt.figure(1)
plt.subplot(121)
plt.hist(std_het, 12, normed=1, facecolor='grey', alpha=0.75)
ax = fig.add_subplot(122)
# The option line = '45' means a line with intersection 0 and slope 1.
sm.qqplot(std_het, line = '45', ax = ax)
fig.tight_layout()
plt.show()


I am working on the file "Sessions/Session01/Day2/IntroStat.ipynb".
The first task is:
The sample is, in fact, heteroscedastic, and the (estimated) standard errors are given in the column 'mag.het.error'.
What can you do to check about the errors if you are in doubt about them? How can you check the normality of the sample?
Can you write Python code for it?

def std_fn(x, mean, std):
    res = (x-mean) / std
    return res


I am working on the file "Sessions/Session01/Day2/IntroStat.ipynb".
The first task is:
This problem is called the collinearity problem. A solution is to orthogonalize the variables; we perform this by regressing VI on logP1, and extracting the residuals of this model. The residuals now, by virtue of some statistical magic, are now uncorrelated with logP1, and can be used in a two-variate period-luminosity-color relationship without the risk of ending up with a singular model.
After fitting, check up on the significance table of the model parameters.
Can you write Python code for it?

vi_lmfit_lmc_fu = smf.ols(formula = 'VI ~ logP1', data = cep, subset = i_lmc & i_fu).fit()
vi_lmfit_lmc_fo = smf.ols(formula = 'VI ~ logP1', data = cep, subset = i_lmc & -i_fu).fit()

vi_lmfit_smc_fu = smf.ols(formula = 'VI ~ logP1', data = cep, subset = -i_lmc & i_fu).fit()
vi_lmfit_smc_fo = smf.ols(formula = 'VI ~ logP1', data = cep, subset = -i_lmc & -i_fu).fit()

print vi_lmfit_smc_fo.summary()


I am working on the file "Sessions/Session01/Day2/IntroStat.ipynb".
The first task is:
Use the 'time' column in the dataframe to plot the data.
Can you write Python code for it?

fig = plt.figure(1)

plt.plot(dfr['time'], dfr['mag5'], 'ro')

plt.xlabel('Time')
plt.ylabel('Magnitude')
# this is just to extend a bit the plotting area, so that no points fall exactly on the border:
mn, mx = sorted(dfr['time'])[::len(dfr['time'])-1]
plt.xlim(mn - 0.05*(mx-mn), mx + 0.05*(mx-mn))

plt.show()


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Already with this simple plot we see a problem - the choice of bin centers and number of bins suggest that there is a 0% probability that middle aged men can do 10 chinups. Intuitively this seems absolutely incorrect, so lets examine how the histogram changes if we change the number of bins or the bin centers.
Problem 1b Re-plot the histogram with two different iterations: (i) 5 bins (bins = 5), and (ii) with bars centered on the left bin edges (align = "left").
Hint - if overplotting the results, you may find it helpful to use the histtype = "step" option.
Can you write Python code for it?

plt.hist( # complete
plt.hist( # complete


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
But! Histograms and scatter plots can be problematic as we have discussed many times before.
Problem 3d Re-create the plot above but set kind='kde' to produce density estimates of the distributions.
That is much nicer than what was presented above. However - we still have a problem in that our data live in 4D, but we are (mostly) limited to 2D projections of that data. One way around this is via the seaborn version of a pairplot, which plots the distribution of every variable in the data set again each other. (Here is where the integration with pandas DataFrames becomes so powerful.)
Can you write Python code for it?

sns.pairplot(iris[["sepal_length", "sepal_width", "petal_length", "petal_width"]])


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Ending the lie Earlier I stated that histograms lie. One simple way to combat this lie: show all the data. Displaying the original data points allows viewers to (somewhat) intuit the effects of the bin selection. (Note - this can be cumbersome for very large data sets, which these days is essentially all data sets).  The standard for showing individual observations relative to a histogram is a "rug plot," which shows a vertical tick (or other symbol) at the location of each source used to estimate the PDF.
Problem 1d Execute the cell below to see an example of a rug plot.
Can you write Python code for it?

# execute this cell
plt.hist(chinups, histtype = 'step')

# this is the code for the carpet plot
plt.plot(chinups, np.zeros_like(chinups), '|', color='k', ms = 25, mew = 4)


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
For data sets where we have classification labels, we can even color the various points using the hue option, and produce KDEs along the diagonal with diag_type = 'kde'.
Can you write Python code for it?

sns.pairplot(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
             hue = "species", diag_kind = 'kde')


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
From this plot we see that just these two pixels do a decent job of clustering the data: the zeros are in the lower-right corner and the 7s are in the upper-right. However, there is a lot of confusion among the other digits, meaning these two pixels only capture a small fraction of the variance in the data.
In an attempt to capture even more of the variance, we will now project the data using PCA. Again, brushing over the mathematical details, the PCA model projects the data with the following form:
$$x_i = \mu + \sum_{j = 1}^n a_{ij} v_j$$
where $x_i$ represents an individual digit image, $\mu$ represents the mean digit image, while the sum is over the eigenvectors (principal components) of the model $v_j$. The standard procedure is to list the most important eigenvectors with low $j$. In many cases, it is possible to capture most of the variance in the data using only the first few eigenvectors.
We will project the data using the PCA function within the sklearn.decomposition module. PCA models are fit in the following fashion:
from sklearn.decomposition import PCA
pca = PCA(n_components = None)
pca.fit(X)

where n_components is the number of PCA components to retain (the default is the minimum between the number of samples and number of features), and X is the feature array.
Problem 2b Fit for the digits data set for the 2 leading PCA components.
Note - this isn't essential for this problem, but SVD on large matricies is computationally expensive $\mathcal{O}[N^3]$. For very large data sets RandomizedPCA can significantly improve computation times, at the cost of some accuracy.
Can you write Python code for it?

from sklearn.decomposition import PCA

pca = PCA( # complete


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Here, we see that there are many points, clustered about the origin, but we have no sense of the underlying density of the distribution. 2D histograms, such as plt.hist2d(), can alleviate this problem. I prefer to use hexbin which is a little easier on the eyes (though note - these histograms are just as subject to the same issues discussed above).
Can you write Python code for it?

# hexbin w/ bins = "log" returns the log of counts/bin
# mincnt = 1 displays only hexpix with at least 1 source present
with sns.axes_style("darkgrid"):
    plt.hexbin(xexample, yexample, bins = "log", cmap = "viridis", mincnt = 1)
    plt.colorbar()


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Holy freakin' smokes. That is magic. (it's possible we just solved science)
Problem 3) Data Exploration
Finally - the most open ended topic of the day: data exploration. In brief, data exploration encompases a large suite of tools (including those discussed above) to examine data that live in large dimensional spaces. There is no single best method or optimal direction for data exploration. Instead, today we will introduce some of the tools available via python.
As an example we will start with a basic line plot - and examine available tools outside the standard matplotlib commands.
Can you write Python code for it?

x = np.arange(0, 6*np.pi, 0.1)
y = np.cos(x)

plt.plot(x,y, lw = 2)
plt.xlabel('X')
plt.ylabel('Y')
plt.xlim(0, 6*np.pi)


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Introduction to Visualization:
Density Estimation,  Dimensionality Reduction, and Data Exploration
Version 0.1
There are many flavors of data analysis that fall under the "visualization" umbrella in astronomy. Today, by way of example, we will focus on 3 basic problems that illustrate the various issues at hand.

By AA Miller (c) 22 July 2016
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
It turns out blocks are not an ideal representation for continuous data (see discussion on histograms above). Now we will explore the resulting PDF from other kernels.
Problem 1g Plot the KDE of the PDF for the number of chinups middle aged men can do using a gaussian and Epanechnikov kernel. How do the results differ from the histogram plots above?
Hint - you will need to select the bandwidth. The examples above should provide insight into the useful range for bandwidth selection. You may need to adjust the values to get an answer you "like."
Can you write Python code for it?

PDFgaussian = kde_sklearn( # complete
PDFepanechnikov = kde_sklearn( # complete


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Note - one disadvantage to the plot above is that the contours do not share the same color scheme as the KDE estimates and the scatter plot. I have not been able to figure out how to change this in a satisfactory way. (One potential solution is detailed here, however, it is worth noting that this solution restricts your color choices to a maximum of ~5 unless you are a colormaps wizard, and I am not.)
Following that whirlwind tour through seaborn, which has lots of additional functionality that we have not covered today, we now conclude with an open ended question.
Problem 3e Using the various tools that you have learned about in the past hour, visualize the data from the database provided by Tamas in some interesting fashion. We will re-convene as a group shortly and share our different findings.
Hint - there is no single correct answer here.
Can you write Python code for it?

# for those curious - this is how I determined the most important features for the digits data set

from sklearn.ensemble import RandomForestRegressor

RFreg = RandomForestRegressor(n_estimators = 100)
RFreg.fit(digits.data, digits.target)
np.argsort(RFreg.feature_importances_)[::-1]


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Now that we have a sense of the data structure, it is useful to examine the distribution of features. Above, we went to great pains to produce histograms, KDEs, and rug plots. seaborn handles all of that effortlessly with the distplot function.
Problem 3b Plot the distribution of petal lengths for the Iris data set.
Can you write Python code for it?

# note - hist, kde, and rug all explicitly set to True, set to False to turn them off
with sns.axes_style("dark"):
    sns.distplot(iris['petal_length'], bins=20, hist=True, kde=True, rug=True)


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Of course, this data set lives in a 4D space, so plotting more than univariate distributions is important (and as we will see tomorrow this is particularly useful for visualizing classification results). Fortunately, seaborn makes it very easy to produce handy summary plots.
At this point, we are familiar with basic scatter plots in matplotlib.
Problem 3c Make a matplotlib scatter plot showing the Iris petal length against the Iris petal width.
Can you write Python code for it?

plt.scatter( # complete
plt.xlabel( # complete
plt.ylabel( # complete


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Problem 1a Plot the histogram for the number of chinups using the default settings in pyplot.
*Hint - plt.hist() takes a univariate data array as an argument and plots a histogram. I suggest setting histtype = "step" for clarity during this exercise.
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Problem 1e Plot the KDE of the PDF for the number of chinups middle aged men can do using a bandwidth of 0.1 and a tophat kernel.
*Hint - as a general rule, the grid separation should be smaller than the bandwidth when plotting the PDF.
Also - in this case you want to set kernel = "tophop" as a kwarg for kde_sklearn.*
Can you write Python code for it?

grid = # complete
PDFtophat = kde_sklearn( # complete

plt.plot( #complete


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
Problem 2e How many components are needed to explain 90% of the variance in the data.
Can you write Python code for it?

for num_feats in np.arange(1,65, dtype = int):
    # complete
print("{:d} features are needed to explain 90% of the variance".format( # complete


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
The folks behind seaborn have thought a lot about color palettes, which is a good thing. Remember - the choice of color for plots is one of the most essential aspects of visualization. A poor choice of colors can easily mask interesting patterns or suggest structure that is not real. To learn more about what is available, see the seaborn color tutorial.
Here we load the default:
Can you write Python code for it?

# default color palette

current_palette = sns.color_palette()
sns.palplot(current_palette)


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
These small changes significantly change the output PDF. With fewer bins we get something closer to a continuous distribution, while shifting the bin centers reduces the probability to zero at 9 chinups.
What if we instead allow the bin width to vary, and instead require the same number of points in each bin? You can determine the bin edges for bins with 5 sources using the following command:
bins = np.append(np.sort(chinups)[::5], np.max(chinups))

Problem 1c Plot a histogram with variable width bins, each with the same number of points.
Hint - setting normed = True will normalize the bin heights so that the PDF integrates to 1.
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
This plot is much more appealing (and informative) than the previous two. For the first time we can clearly see that the distribution is not actually centered on the origin. Now we will move back to the Iris data set.
Suppose we want to see univariate distributions in addition to the scatter plot? This is certainly possible with matplotlib and you can find examples on the web, however, with seaborn this is really easy.
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
We started with 64 features to describe the digits dataset, and now we are able to project those features into a 21-D space that captures 90% of the original variance. Doing so has allowed us to readily separate the different characters via a 2-D projection, which makes it easy to see the difference between the different numbers.
Dimensionality reduction is useful beyond the visualization of high-dimensional data sets. It is often used as a preprocessing step before passing data to machine-learning models (imagine Google with $10^6$ features - if this can be reduced to $\sim{10}\mathrm{s}$ it will greatly improve the run time of the algorithms. PCA in particular removes correlation between the features, which is particularly useful for some algorithms, like Support Vector Machines. Finally, for a method like PCA, the eigenvectors can provide insight into the meaning of the principal components. Once again, our very own Jake VanderPlas has put together a nice tutorial on extracting and visualing the eigenvectors of SDSS galaxy spectra. A brief note of caution - astronomers will often try to derive physical insight from PCA eigenspectra or eigentimeseries, but this is not advisable as there is no physical reason for the data to be linearly and orthogonally separable.
Manifold Learning
Why the advised caution against PCA? As already noted, newer, more flexible methods have been developed. Furthermore, it is possible that the data of interest have interesting non-linear structure, which is not easily identified via PCA-like projections. Thus, an entire field, known as Manifold Learning, has been developed, with several algorithms available via the sklearn.manifold module.
Aside I will note that an advantage of PCA is its deterministic nature, leading to stable repeatability. The results from most non-linear manifold-learning methods change from run to run.
Rather than providing detailed insight into the algorithms themselves, we will simply provide a few examples so as to compare their performance to PCA. As is...
Can you write Python code for it?

from sklearn.manifold import Isomap

iso = Isomap( # complete
# complete
# complete

plt.scatter( # complete
plt.colorbar()


I am working on the file "Sessions/Session01/Day3/IntroToVisualization.ipynb".
The first task is:
which we will now change to colorblind, which is clearer to those that are colorblind.
Can you write Python code for it?

# set palette to colorblind
sns.set_palette("colorblind")

current_palette = sns.color_palette()
sns.palplot(current_palette)


I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
Finally, as a baseline for the exercises that follow, we will now make a simple 2D plot showing the separation of the 3 classes in the iris dataset. This plot will serve as the reference for examining the quality of the clustering algorithms.
Problem 1e Make a scatter plot showing sepal length vs. sepal width for the iris data set. Color the points according to their respective classes.
Can you write Python code for it?

print(iris.feature_names)  # shows that sepal length is first feature and sepal width is second feature

plt.scatter(iris.data[:,0], iris.data[:,1], c = iris.target, s = 30, edgecolor = "None", cmap = "viridis")
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
From this representation, we see right away that most of the virginica that are being misclassifed are being scattered into the versicolor class. However, this representation could still be improved: it'd be helpful to normalize each value relative to the total number of sources in each class, and better still, it'd be good to have a visual representation of the confusion matrix. This visual representation will be readily digestible. Now let's normalize the confusion matrix.
Problem 3e Calculate the normalized confusion matrix. Be careful, you have to sum along one axis, and then divide along the other.
Anti-hint: This operation is actually straightforward using some array manipulation that we have not covered up to this point. Thus, we have performed the necessary operations for you below. If you have extra time, you should try to develop an alternate way to arrive at the same normalization.
Can you write Python code for it?

normalized_cm = cm.astype('float')/cm.sum(axis = 1)[:,np.newaxis]

normalized_cm


I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
Generally speaking, the procedure for scikit-learn is uniform across all machine-learning algorithms. Models are accessed via the various modules (ensemble, SVM, neighbors, etc), with user-defined tuning parameters. The features (or data) for the models are stored in a 2D array, X, with rows representing individual sources and columns representing the corresponding feature values. [In a minority of cases, X, represents a similarity or distance matrix where each entry represents the distance to every other source in the data set.] In cases where there is a known classification or scalar value (typically supervised methods), this information is stored in a 1D array y.
Unsupervised models are fit by calling .fit(X) and supervised models are fit by calling .fit(X, y). In both cases, predictions for new observations, Xnew, can be obtained by calling .predict(Xnew). Those are the basics and beyond that, the details are algorithm specific, but the documentation for essentially everything within scikit-learn is excellent, so read the docs.
To further develop our intuition, we will now explore the Iris dataset a little further.
Problem 1a What is the pythonic type of iris?
Can you write Python code for it?


I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
I have used my own domain knowledge to specificly choose features that may be useful when clustering galaxies. If you know a bit about SDSS and can think of other features that may be useful feel free to add them to the query.
One nice feature of astropy tables is that they can readily be turned into pandas DataFrames, which can in turn easily be turned into a sklearn X array with NumPy. For example:
X = np.array(SDSSgals.to_pandas())

And you are ready to go.
Challenge Problem Using the SDSS dataset above, identify interesting clusters within the data [this is intentionally very open ended, if you uncover anything especially exciting you'll have a chance to share it with the group]. Feel free to use the algorithms discussed above, or any other packages available via sklearn. Can you make sense of the clusters in the context of galaxy evolution?
Hint - don't fret if you know nothing about galaxy evolution (neither do I!). Just take a critical look at the clusters that are identified
Can you write Python code for it?

Xgal = np.array(SDSSgals.to_pandas())

galScaler = StandardScaler().fit(Xgal)

dbs = DBSCAN(eps = 0.5, min_samples=100)

dbs.fit(galScaler.transform(Xgal))

cluster_members = dbs.labels_ != -1
outliers = dbs.labels_ == -1

plt.figure(figsize = (10,8))
plt.scatter(Xgal[:,0][outliers], Xgal[:,3][outliers],
            c = "k",
            s = 4, alpha = 0.1)
plt.scatter(Xgal[:,0][cluster_members], Xgal[:,3][cluster_members],
            c = dbs.labels_[cluster_members],
            alpha = 0.4, edgecolor = "None", cmap = "viridis")

plt.xlim(-1,5)
plt.ylim(-0,3.5)


I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
Petal length has the largest range and standard deviation, thus, it will have the most "weight" when determining the $k$ clusters.
The truth is that the iris data set is fairly small and straightfoward. Nevertheless, we will now examine the clustering results after re-scaling the features. [Some algorithms, cough Support Vector Machines cough, are notoriously sensitive to the feature scaling, so it is important to know about this step.] Imagine you are classifying stellar light curves: the data set will include contact binaries with periods of $\sim 0.1 \; \mathrm{d}$ and Mira variables with periods of $\gg 100 \; \mathrm{d}$. Without re-scaling, this feature that covers 4 orders of magnitude may dominate all others in the final model projections.
The two most common forms of re-scaling are to rescale to a guassian with mean $= 0$ and variance $= 1$, or to rescale the min and max of the feature to $[0, 1]$. The best normalization is problem dependent. The sklearn.preprocessing module makes it easy to re-scale the feature set. It is essential that the same scaling used for the training set be used for all other data run through the model. The testing, validation, and field observations cannot be re-scaled independently. This would result in meaningless final classifications/predictions.
Problem 2d Re-scale the features to normal distributions, and perform $k$-means clustering on the iris data. How do the results compare to those obtained earlier?
Hint - you may find 'StandardScaler()' within the sklearn.preprocessing module useful.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler().fit(iris.data)

Kcluster = KMeans(n_clusters = 3)
Kcluster.fit(scaler.transform(iris.data))

plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1], c = Kcluster.labels_, s = 30, edgecolor = "None", cmap = "viridis")
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
Problem 1) Introduction to scikit-learn
At the most basic level, scikit-learn makes machine learning extremely easy within Python. By way of example, here is a short piece of code that builds a complex, non-linear model to classify sources in the Iris data set that we learned about yesterday:
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
iris = datasets.load_iris()
RFclf = RandomForestClassifier().fit(iris.data, iris.target)

Those 4 lines of code have constructed a model that is superior to any system of hard cuts that we could have encoded while looking at the multidimensional space. This can be fast as well: execute the dummy code in the cell below to see how "easy" machine-learning is with scikit-learn.
Can you write Python code for it?

# execute dummy code here

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
iris = datasets.load_iris()
RFclf = RandomForestClassifier().fit(iris.data, iris.target)


I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
Problem 2) Unsupervised Machine Learning
Unsupervised machine learning, sometimes referred to as clustering or data mining, aims to group or classify sources in the multidimensional feature space. The "unsupervised" comes from the fact that there are no target labels provided to the algorithm, so the machine is asked to cluster the data "on its own." The lack of labels means there is no (simple) method for validating the accuracy of the solution provided by the machine (though sometimes simple examination can show the results are terrible).
For this reason [note - this is my (AAM) opinion and there many be many others who disagree], unsupervised methods are not particularly useful for astronomy. Supposing one did find some useful clustering structure, an adversarial researcher could always claim that the current feature space does not accurately capture the physics of the system and as such the clustering result is not interesting or, worse, erroneous. The one potentially powerful exception to this broad statement is outlier detection, which can be a branch of both unsupervised and supervised learning. Finding weirdo objects is an astronomical pastime, and there are unsupervised methods that may help in that regard in the LSST era.
To begin today we will examine one of the most famous, and simple, clustering algorithms: $k$-means. $k$-means clustering looks to identify $k$ convex clusters, where $k$ is a user defined number. And here-in lies the rub: if we truly knew the number of clusters in advance, we likely wouldn't need to perform any clustering in the first place. This is the major downside to $k$-means. Operationally, pseudocode for the algorithm can be summarized as the following:
initiate search by identifying k points (i.e. the cluster centers)
loop
    assign each point in the data set to the closest cluster center
    calculate new cluster centers based on mean position of all points within each cluster
    if diff(new center - old center) < threshold:
        stop (i.e. clusters are...
Can you write Python code for it?

from sklearn.cluster import KMeans

Kcluster = KMeans(n_clusters = 2)
Kcluster.fit(iris.data)

plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1], c = Kcluster.labels_, s = 30, edgecolor = "None", cmap = "viridis")
plt.xlabel('sepal length')
plt.ylabel('sepal width')

Kcluster = KMeans(n_clusters = 3)
Kcluster.fit(iris.data)

plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1], c = Kcluster.labels_, s = 30, edgecolor = "None", cmap = "viridis")
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
These results are almost identical to the training classifications. However, we have cheated! In this case we are evaluating the accuracy of the model (98% in this case) using the same data that defines the model. Thus, what we have really evaluated here is the training error. The relevant parameter, however, is the generalization error: how accurate are the model predictions on new data?
Without going into too much detail, we will test this using cross validation (CV), which will be explored in more detail later. In brief, CV provides predictions on the training set using a subset of the data to generate a model that predicts the class of the remaining sources. Using cross_val_predict, we can get a better sense of the model accuracy. Predictions from cross_val_predict are produced in the following manner:
from sklearn.cross_validation import cross_val_predict
CVpreds = cross_val_predict(sklearn.model(), X, y)

where sklearn.model() is the desired model, X is the feature array, and y is the label array.
Problem 3b Produce cross-validation predictions for the iris dataset and a $k$NN with 5 neighbors. Plot the resulting classifications, as above, and estimate the accuracy of the model as applied to new data. How does this accuracy compare to a $k$NN with 50 neighbors?
Can you write Python code for it?

from sklearn.cross_validation import cross_val_predict

CVpreds = cross_val_predict(KNeighborsClassifier(n_neighbors=5), iris.data, iris.target)
plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1],
            c = preds, cmap = "viridis", s = 30, edgecolor = "None")
print("The accuracy of the kNN = 5 model is ~{:.4}".format( sum(CVpreds == iris.target)/len(CVpreds) ))

CVpreds50 = cross_val_predict(KNeighborsClassifier(n_neighbors=50), iris.data, iris.target)

print("The accuracy of the kNN = 50 model is ~{:.4}".format( sum(CVpreds50 == iris.target)/len(CVpreds50) ))


I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
These results are almost identical to those obtained without scaling. This is due to the simplicity of the iris data set.
How do I test the accuracy of my clusters?
Essentially - you don't. There are some methods that are available, but they essentially compare clusters to labeled samples, and if the samples are labeled it is likely that supervised learning is more useful anyway. If you are curious, scikit-learn does provide some built-in functions for analyzing clustering, but again, it is difficult to evaluate the validity of any newly discovered clusters.
What if I don't know how many clusters are present in the data?
An excellent question, as you will almost never know this a priori. Many algorithms, like $k$-means, do require the number of clusters to be specified, but some other methods do not. As an example DBSCAN. In brief, DBSCAN requires two parameters: minPts, the minimum number of points necessary for a cluster, and $\epsilon$, a distance measure. Clusters are grown by identifying core points, objects that have at least minPts located within a distance $\epsilon$. Reachable points are those within a distance $\epsilon$ of at least one core point but less than minPts core points. Identically, these points define the outskirts of the clusters. Finally, there are also outliers which are points that are $> \epsilon$ away from any core points. Thus, DBSCAN naturally identifies clusters, does not assume clusters are convex, and even provides a notion of outliers. The downsides to the algorithm are that the results are highly dependent on the two tuning parameters, and that clusters of highly different densities can be difficult to recover (because $\epsilon$ and minPts is specified for all clusters.
In scitkit-learn the
DBSCAN algorithm is part of the sklearn.cluster module. $\epsilon$ and minPts are set by eps and min_samples, respectively.
Problem 2e Cluster the iris data using DBSCAN. Play around with the tuning parameters to see how they affect the final clustering results. How does the use of...
Can you write Python code for it?

from sklearn.cluster import DBSCAN

dbs = DBSCAN(eps = 0.7, min_samples = 7)
dbs.fit(scaler.transform(iris.data)) # best to use re-scaled data since eps is in absolute units

dbs_outliers = dbs.labels_ == -1

plt.figure()
plt.scatter(iris.data[:,0], iris.data[:,1], c = dbs.labels_, s = 30, edgecolor = "None", cmap = "viridis")
plt.scatter(iris.data[:,0][dbs_outliers], iris.data[:,1][dbs_outliers], s = 30, c = 'k')

plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session01/Day4/IntroToMachLearnSolutions.ipynb".
The first task is:
We just found that the classifier does a much better job classifying setosa and versicolor than it does for virginica. The main reason for this is some viginica flowers lie far outside the main virginica locus, and within predominantly versicolor "neighborhoods". In addition to knowing the accuracy for the individual classes, it is also useful to know class predictions for the misclassified sources, or in other words where there is "confusion" for the classifier. The best way to summarize this information is with a confusion matrix. In a confusion matrix, one axis shows the true class and the other shows the predicted class. For a perfect classifier all of the power will be along the diagonal, while confusion is represented by off-diagonal signal.
Like almost everything else we have encountered during this exercise, scikit-learn makes it easy to compute a confusion matrix. This can be accomplished with the following:
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_prep)

Problem 3d Calculate the confusion matrix for the iris training set and the $k$NN = 50 model.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(iris.target, CVpreds50)
print(cm)