I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
Boolean Type
The Boolean type is a simple type with two possible values: True and False, and is returned by comparison operators discussed previously:
Can you write Python code for it?


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
Complex Numbers
Complex numbers are numbers with real and imaginary (floating-point) parts.
We've seen integers and real numbers before; we can use these to construct a complex number:
Can you write Python code for it?


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
Complex numbers have a variety of interesting attributes and methods, which we'll briefly demonstrate here:
Can you write Python code for it?

c = 3 + 4j

c.real  # real part

c.imag  # imaginary part

c.conjugate()  # complex conjugate

abs(c)  # magnitude, i.e. sqrt(c.real ** 2 + c.imag ** 2)


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
Floating-Point Numbers
The floating-point type can store fractional numbers.
They can be defined either in standard decimal notation, or in exponential notation:
Can you write Python code for it?


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
In the exponential notation, the e or E can be read "...times ten to the...",
so that 1.4e6 is interpreted as $~1.4 \times 10^6$.
An integer can be explicitly converted to a float with the float constructor:
Can you write Python code for it?


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
None Type
Python includes a special type, the NoneType, which has only a single possible value: None. For example:
Can you write Python code for it?


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
Python has many extremely useful string functions and methods; here are a few of them:
Can you write Python code for it?

# length of string
len(response)

# Make upper-case. See also str.lower()
response.upper()

# Capitalize. See also str.title()
message.capitalize()

# multiplication is multiple concatenation
5 * response

# concatenation with +
message + response


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
Q: Now try to split this notebook's name into words. What argument you have to pass to split()?
Can you write Python code for it?

notebook_name = '05-Built-in-Scalar-Types.ipynb'

notebook_name.split()


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
Q: When you have a sequence of string, you can use join() method:
Can you write Python code for it?

l = notebook_name.split('-')
' '.join(l)


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
String Type
Strings in Python are created with single or double quotes:
Can you write Python code for it?

message = "what do you like?"
response = 'spam'


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
String formatting

by order
Can you write Python code for it?


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
You will know more about lists in Python in the next notebook.
Indexing
Can you write Python code for it?

# Access individual characters (zero-based indexing)
message[0:5]

message[5:11]

message[11:17] # note that spaces count as a character


I am working on the file "Day_1/05-Built-in-Scalar-Types.ipynb".
The first task is:
You'll see None used in many places, but perhaps most commonly it is used as the default return value of a function.
For example, the print() function in Python 3 does not return anything, but we can still catch its value:
Can you write Python code for it?

return_value = print('abc')

print(return_value)


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
A particularly useful version of this is to specify a negative step, which will reverse the array:
Can you write Python code for it?


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
A very similar slicing syntax is also used in other data containers, such as NumPy arrays, as we will see in Day 2 sessions.
Tuples
Tuples are in many ways similar to lists, but they are defined with parentheses and cannot be changed!
Can you write Python code for it?


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
Both indexing and slicing can be used to set elements as well as access them.
The syntax is as you would expect:
Can you write Python code for it?


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
If you're familiar with the mathematics of sets, you'll be familiar with operations like the union, intersection, difference, symmetric difference, and others.
Python's sets have all of these operations built-in, via methods or operators.
For each, we'll show the two equivalent methods:
Can you write Python code for it?

# union: items appearing in either
primes | odds      # with an operator
primes.union(odds) # equivalently with a method

# intersection: items appearing in both
primes & odds             # with an operator
primes.intersection(odds) # equivalently with a method

# difference: items in primes but not in odds
primes - odds           # with an operator
primes.difference(odds) # equivalently with a method

# symmetric difference: items appearing in only one set
primes ^ odds                     # with an operator
primes.symmetric_difference(odds) # equivalently with a method


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
Keep in mind that dictionaries do not maintain any order for the input parameters.
Sets
The 4th basic data container is the set, which contains unordered collections of unique items.
They are defined much like lists and tuples, except they use the curly brackets of dictionaries.
They do not contain duplicate entries. Which means they are significantly faster than lists!
http://stackoverflow.com/questions/2831212/python-sets-vs-lists
Can you write Python code for it?


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
Lists have a number of useful properties and methods available to them.
Can you write Python code for it?

# Length of a list
len(a)

# Append a value to the end
a.append(11)
a

# Addition concatenates lists
a + [13, 17, 19]

# sort() method sorts in-place
a = [2, 5, 1, 6, 3, 4]
a.sort()
a


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
New items can be added to the dictionary using indexing as well:
Can you write Python code for it?

# Set a new key:value pair
numbers['ninety'] = 90
print(numbers)


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
Similarly, if we leave out the last index, it defaults to the length of the list.
Thus, the last three elements can be accessed as follows:
Can you write Python code for it?


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
The main distinguishing feature of tuples is that they are immutable: this means that once they are created, their size and contents cannot be changed:
Tuples are often used in a Python program; e.g. in functions that have multiple return values.
Dictionaries
Dictionaries are extremely flexible mappings of keys to values, and form the basis of much of Python's internal implementation.
They can be created via a comma-separated list of key:value pairs within curly braces:
Can you write Python code for it?

numbers = {'one':1, 'two':2, 'three':3}
# or
numbers = dict(one=1, two=2, three=2)


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
They can also be defined without any brackets at all:
Can you write Python code for it?


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
You can visualize this indexing scheme this way:

Here values in the list are represented by large numbers in the squares; list indices are represented by small numbers above and below.
In this case, L[2] returns 5, because that is the next value at index 2.
Where indexing is a means of fetching a single value from the list, slicing is a means of accessing multiple values in sub-lists.
It uses a colon to indicate the start point (inclusive) and end point (non-inclusive) of the sub-array.
For example, to get the first three elements of the list, we can write:
Can you write Python code for it?


I am working on the file "Day_1/06-Built-in-Data-Structures.ipynb".
The first task is:
we can equivalently write:
Can you write Python code for it?


I am working on the file "Day_1/07-Ex-Try-Python-Basics.ipynb".
The first task is:
3. Mutability of lists and tuples
Can you write Python code for it?

s = (1, 2, 3) # tuple

s[1] = 4 # Can you remember that tuples cannot be changed!

s.append(4) # this is never going to work as it is a tuple!
s

t = [1, 2, 3] # But a list might just do the job:

t[1] = 4

t.append(4)
t


I am working on the file "Day_1/07-Ex-Try-Python-Basics.ipynb".
The first task is:
4. Dictionary attributes
Can you write Python code for it?

foo = dict(a='123', say='hellozles', other_key=['wow', 'a', 'list', '!'], and_another_key=3.14)

# foo.update(dict(a=42))
## or
# foo.update(a=42)

# foo.pop('say')

# foo.items()

# foo.keys()

# foo.values()


I am working on the file "Day_1/07-Ex-Try-Python-Basics.ipynb".
The first task is:
For sequences, which we'll see in the next section, the Boolean representation is False for empty sequences and True for any other sequences
Can you write Python code for it?


I am working on the file "Day_1/07-Ex-Try-Python-Basics.ipynb".
The first task is:
For strings, bool(s) is False for empty strings and True otherwise:
Can you write Python code for it?


I am working on the file "Day_1/07-Ex-Try-Python-Basics.ipynb".
The first task is:
Python internally truncates these representations at 52 bits beyond the first nonzero bit on most systems.

This rounding error for floating-point values is a necessary evil of working with floating-point numbers.

The best way to deal with it is to always keep in mind that floating-point arithmetic is approximate, and never rely on exact equality tests with floating-point values.

2. Explore Booleans
Booleans can also be constructed using the bool() object constructor: values of any other type can be converted to Boolean via predictable rules.
For example, any numeric type is False if equal to zero, and True otherwise:
Can you write Python code for it?


I am working on the file "Day_1/07-Ex-Try-Python-Basics.ipynb".
The first task is:
Why is this the case? It turns out that it is not a behavior unique to Python, but is due to the fixed-precision format of the binary floating-point storage.
All programming languages using floating-point numbers store them in a fixed number of bits, and this leads some numbers to be represented only approximately.
We can see this by printing the three values to high precision:
Can you write Python code for it?


I am working on the file "Day_1/08-Jupyter-Notebook.ipynb".
The first task is:
2. Inline Documentation
Can you write Python code for it?


I am working on the file "Day_1/08-Jupyter-Notebook.ipynb".
The first task is:
3. Tab Completion (predictive and helpful!)
Type o then tab and you should see a drop-down menu of choices.
Can you write Python code for it?

#o     # Always press the tab! Tab is the same as Unix. It works a little like predictive text messaging if you are unfamilar with it.


I am working on the file "Day_1/08-Jupyter-Notebook.ipynb".
The first task is:
7. Load Remote Code - This is Very Cool and helpful!
You can pull up remote code using loadpy
python
loadpy http://matplotlib.org/mpl_examples/mplot3d/2dcollections3d_demo.py
Can you write Python code for it?


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
Although this does work, Python provides a cleaner syntax using the enumerate iterator:
Can you write Python code for it?

for i, val in enumerate(L):
    print(i, val)


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
Any number of iterables can be zipped together, and if they are different lengths, the shortest will determine the length of the zip.
map(function, iterable, ...)
Apply function to every item of iterable and return a list of the results.
Can you write Python code for it?


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
Had we not thrown-in a loop break here, it would go on happily counting until the process is manually interrupted or killed (using, for example, ctrl-C).
If range were to actually create that list of one trillion values, it would occupy tens of GB of machine memory: a waste, given the fact that we're ignoring all but the first 10 values!
Useful Iterators
enumerate
Often you need to iterate not only the values in an array, but also keep track of the index.
You might be tempted to do things this way:
Can you write Python code for it?

L = [2, 4, 6, 8, 10]
for i in range(len(L)):
    print(i, L[i])


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
If range were to actually create that list of one trillion values, it would occupy tens of terabytes of machine memory: a waste, given the fact that we're ignoring all but the first 10 values!
In fact, there's no reason that iterators ever have to end at all! Python's itertools library contains a count function that acts as an infinite range:
Can you write Python code for it?

from itertools import count

for i in count():
    if i >= 10:
        break
    print(i, end=', ')


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
Note especially the use of colons (:) and whitespace to denote separate blocks of code.
for loops
Loops in Python are a way to repeatedly execute some code statement.
This saves time rather than writing out each line, but can also be computationally demanding!
So, for example, if we'd like to print each of the items in a list, we can use a for loop:
Can you write Python code for it?


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
Now lets print it on the same line for ease:
A 'for loop' is a basic and extremely helpful tool in python.
What we are doing here is replace the 'N' with each number within the list ie. [2,3,5,7] sequentionally, and printing the output.
For each number you printed, you can see it has been placed on a new line. This signifies the loop has restarted on the next value in the list.
Now let's do the same but this time put all the numbers on the same line for visual appearance.
Can you write Python code for it?

for N in [2, 3, 5, 7]: # you can replace 'N' with anything you like, but make sure to replace it below too!
    print(N, end=' ') # print all on same line


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
Range objects can also have more complicated values:
Can you write Python code for it?

# range from 5 to 10
list(range(5, 10))

# range from 0 to 10 by 2
list(range(0, 10, 2))


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
References
A Whirlwind Tour of Python by Jake VanderPlas (O’Reilly). Copyright 2016 O’Reilly Media, Inc., 978-1-491-96465-1
Solutions to Exercise 1
Can you write Python code for it?

N=5
#Solution
for i in range(N):
    for j in range(i+1):
        print ('*', end=' ')
    print('')
print('')

#Solution using break
for i in range(N):
    for j in range(N):
        if (j > i):
            break
        print('*', end=' ')
    print('')


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
The argument of the while loop is evaluated as a boolean statement, and the loop is executed until the statement evaluates to False.
break and continue: Fine-tuning your For and  While loops

break statement breaks-out of the loop entirely
continue statement skips the remainder of the current loop, and goes to the next iteration

Let's print a string of odd numbers.
Can you write Python code for it?

for n in range(20):
    # check if n is even
    if n % 2 == 0:
        continue
    print(n,end=" ")


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
itertools
More specialised iterators are available in a built-in itertools module. Example:
Can you write Python code for it?

from itertools import permutations
p = permutations(range(3))
print(*p)


I am working on the file "Day_1/09-Control-Flow-Statements.ipynb".
The first task is:
range, like a list, exposes an iterator:
Can you write Python code for it?


I am working on the file "Day_1/10-Defining-Functions.ipynb".
The first task is:
*args and **kwargs: Flexible Arguments
How about a a function which you don't initially know how many arguments there are?
We can use the special form *args (arguments) and **kwargs (keyword arguments)to catch all the arguments that are passed.
Here is an example:
Can you write Python code for it?

def catch_all(*args, **kwargs):
    print("args =", args)
    print("kwargs = ", kwargs)

catch_all(1, 2, 3, python=4, conda=5)

catch_all('a', keyword=2)


I am working on the file "Day_1/10-Defining-Functions.ipynb".
The first task is:
But dictionaries are not orderable: we need a way to tell the function how to sort our data.
We can do this by specifying the key function, a function which given an item returns the sorting key for that item:
Can you write Python code for it?

# sort alphabetically by first name
sorted(data, key=lambda item: item['first'])

# sort by year of birth
sorted(data, key=lambda item: item['YOB'])


I am working on the file "Day_1/10-Defining-Functions.ipynb".
The first task is:
Here it is not the names args and kwargs that are important, but the * characters preceding them.
args and kwargs are just the variable names often used by convention, short for "arguments" and "keyword arguments".
The operative difference is the asterisk characters: a single * before a variable means "expand this as a sequence", while a double ** before a variable means "expand this as a dictionary".
Can you write Python code for it?

inputs = (1, 2, 3)
keywords = {'pi': 3.14}

catch_all(*inputs, **keywords)

print()
print('compare to this:')
print()

catch_all(inputs, keywords)


I am working on the file "Day_1/10-Defining-Functions.ipynb".
The first task is:
Note: printing data out does not return it to you as output. To get output from a function, you have to use return statement:
Can you write Python code for it?

def square_fun(x):
    return x**2

result = square_fun(4)

result


I am working on the file "Day_1/10-Defining-Functions.ipynb".
The first task is:
Now suppose we want to sort this data.
Python has a sorted function that does this:
Can you write Python code for it?


I am working on the file "Day_1/10-Defining-Functions.ipynb".
The first task is:
Now we have a function named fibonacci which takes a single argument N, does something with this argument, and returns a value; in this case, a list of the first N Fibonacci numbers:
Can you write Python code for it?

fibonacci(10)
# Now we can call the function by typing the name, and putting a value of our choice into the function using brackets.


I am working on the file "Day_1/10-Defining-Functions.ipynb".
The first task is:
Put the name of an animal into the brackets. Hint: be sure to make it a string!
Can you write Python code for it?


I am working on the file "Day_1/10-Defining-Functions.ipynb".
The first task is:
The values can also be specified by name if desired, in which case the order of the named values does not matter:
Can you write Python code for it?


I am working on the file "Day_1/10-Defining-Functions.ipynb".
The first task is:
This lambda function is roughly equivalent to
Can you write Python code for it?


I am working on the file "Day_1/13-Ex-Create-module.ipynb".
The first task is:
5. Create a dictionary of data, whose keys are names of regions and values are lists of heat fluxes data

Create a list of names of the regions/experiments
Create an empty dictionary, named hf_dict or whatever sounds better to you
Loop over the names, call the create_sst() function and assign it to a variable, e.g. fake_sst
Still inside the name-loop, write another loop to iterate over SST and wind values, just as you did in the previous exercise, and calculate the heat flux.
Assign the result to the corresponding key of hf_dict
Can you write Python code for it?


I am working on the file "Day_1/13-Ex-Create-module.ipynb".
The first task is:
Bonus
Can you write Python code for it?


I am working on the file "Day_1/13-Ex-Create-module.ipynb".
The first task is:
Print the result to test yourself.
6. Save the dictionary to text file (bonus: to a json file), both keys and values

You can copy the code for writing data to a text file from the previous exercise
Modify it so that the output file would include hf_dict's keys as row (column) names
Can you write Python code for it?


I am working on the file "Day_1/13-Ex-Create-module.ipynb".
The first task is:
or
Can you write Python code for it?

# from my_super_awesome_module import heat_flux_function, create_sst


I am working on the file "Day_1/solutions/11-Ex-Build-A-Program-Solution.ipynb".
The first task is:
4. Open a new text file for writing
Now, you need to open a file. Explore the built-in function open():
Can you write Python code for it?


I am working on the file "Day_1/solutions/11-Ex-Build-A-Program-Solution.ipynb".
The first task is:
Print out heat_flux variable to check that the values are sensible (no pun intended).
Can you write Python code for it?


I am working on the file "Day_1/solutions/11-Ex-Build-A-Program-Solution.ipynb".
The first task is:
Print out the wind_speed and sst variables to check yourself.
Can you write Python code for it?


I am working on the file "Day_1/solutions/11-Ex-Build-A-Program-Solution.ipynb".
The first task is:
Use a text editor of your choice (or Jupyter!) to check the contents of the file.
Can you write Python code for it?


I am working on the file "Day_1/solutions/13-Ex-Create-module-Solution.ipynb".
The first task is:
5. Create a dictionary of data, whose keys are names of regions and values are lists of heat fluxes data

Create a list of names of the regions/experiments
Create an empty dictionary, named hf_dict or whatever sounds better to you
Loop over the names, call the create_sst() function and assign it to a variable, e.g. fake_sst
Still inside the name-loop, write another loop to iterate over SST and wind values, just as you did in the previous exercise, and calculate the heat flux.
Assign the result to the corresponding key of hf_dict
Can you write Python code for it?

regions = ['WS', 'BS']
hf_dict = dict()
for reg in regions:
    fake_sst = create_sst(reg)
    heat_flux = []
    for u, t in zip(wind_speed, fake_sst):
        q = calc_heat_flux(u, t)
        heat_flux.append(q)
    hf_dict[reg] = heat_flux


I am working on the file "Day_1/solutions/13-Ex-Create-module-Solution.ipynb".
The first task is:
6. Save the dictionary to text file, both keys and values

You can copy the code for writing data to a text file from the previous exercise
Modify it so that the output file would include hf_dict's keys as row (column) names
Can you write Python code for it?

with open('heat_flux_var_sst_bycol.txt', 'w') as f:
    column_names = sorted(hf_dict.keys())
    f.write(','.join(column_names)+'\n')
    for tup in zip(*[hf_dict[i] for i in column_names]):
        f.write(','.join([str(i) for i in tup])+'\n')

with open('heat_flux_var_sst_byrow.txt', 'w') as f:
    for k, v in hf_dict.items():
        line = k + ','
        for i in v:
            line += str(i) + ','
        line = line[:-1]+ '\n'
        f.write(line)

!more {f.name}


I am working on the file "Day_1/solutions/13-Ex-Create-module-Solution.ipynb".
The first task is:
Print the result to test yourself.
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
3. Data import and export
A wide range of input/output formats are natively supported by pandas:

CSV, text
SQL database
Excel
HDF5
json
html
pickle
sas, stata
(parquet)
...
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
4. Exploration
Some useful methods:
head and tail
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
6. The group-by operation
Some 'theory': the groupby operation (split-apply-combine)
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Based on the titanic data set, how many passengers older than 70 were on the Titanic?

Can you write Python code for it?

# %load snippets/02-pandas_introduction67.py

# %load snippets/02-pandas_introduction68.py


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Based on the titanic data set, select all rows for male passengers and calculate the mean age of those passengers. Do the same for the female passengers

Can you write Python code for it?

df = pd.read_csv("data/titanic.csv")

# %load snippets/02-pandas_introduction64.py

# %load snippets/02-pandas_introduction65.py

# %load snippets/02-pandas_introduction66.py


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Calculate the average survival ratio for all passengers.

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Calculate this survival ratio for all passengers younger that 25 (remember: filtering/boolean indexing).

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

What is the difference in the survival ratio between the sexes?

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

What is the maximum Fare that was paid? And the median?

Can you write Python code for it?

# %load snippets/02-pandas_introduction31.py

# %load snippets/02-pandas_introduction32.py


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Plot the age distribution of the titanic passengers

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:

EXERCISE: The evolution of the yearly averages with, and the overall mean of all stations

Use `resample` and `plot` to plot the yearly averages for the different stations.

The overall mean of all stations can be calculated by taking the mean of the different columns (`.mean(axis=1)`).

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:

EXERCISE: The typical diurnal profile for the different stations

Similar as for the month, you can now group by the hour of the day.

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:

EXERCISE: What is the difference in the typical diurnal profile between week and weekend days for the 'BASCH' station.

Add a column 'weekday' defining the different days in the week.

Add a column 'weekend' defining if a days is in the weekend (i.e. days 5 and 6) or not (True/False).

You can groupby on multiple items at the same time. In this case you would need to group by both weekend/weekday and hour of the day.

Add a column indicating the weekday:
Can you write Python code for it?

no2.index.weekday?

# %load snippets/02-pandas_introduction103.py


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
A nice feature is "partial string" indexing, so you don't need to provide the full datetime string.
E.g. all data of January up to March 2012:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
An overview of that information can be given with the info() method:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Application of the groupby concept on the titanic data
We go back to the titanic passengers survival data:
Can you write Python code for it?

df = pd.read_csv("data/titanic.csv")

df.head()


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Attributes of a Series: index and values
The Series has also an index and values attribute, but no columns
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
But also a lot of pandas specific methods, e.g.
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
But, slicing accesses the rows:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Getting some basic summary statistics about the data with describe:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
However, in many cases your data has certain groups in it, and in that case, you may want to calculate this statistic for each of the groups.
For example, in the above dataframe df, there is a column 'key' which has three possible values: 'A', 'B' and 'C'. When we want to calculate the sum for each of those groups, we could do the following:
Can you write Python code for it?

for key in ['A', 'B', 'C']:
    print(key, df[df['key'] == key]['data'].sum())


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Indexing a time series works with strings:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Luckily, if we have a well formed csv file, we don't need many of those arguments:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Now we can groupby the hour of the day and the weekend (or use pivot_table):
Can you write Python code for it?

# %load snippets/02-pandas_introduction105.py

# %load snippets/02-pandas_introduction106.py

# %load snippets/02-pandas_introduction107.py

# %load snippets/02-pandas_introduction108.py


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Now, we can calculate the mean of each month over the different years:
Can you write Python code for it?

# %load snippets/02-pandas_introduction99.py

# %load snippets/02-pandas_introduction100.py


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
One-dimensional data: Series (a column of a DataFrame)
A Series is a basic holder for one-dimensional labeled data.
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Or how does it differ between the different classes?
Can you write Python code for it?

df.groupby('Pclass')['Survived'].aggregate(lambda x: x.sum() / len(x)).plot(kind='bar')


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Or we can use some more advanced time series features -> see further in this notebook!
5. Selecting and filtering data

ATTENTION!:

One of pandas' basic features is the labeling of rows and columns, but this makes indexing also a bit more complex compared to numpy.

 We now have to distuinguish between:

selection by **label**

selection by **position**

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Quickly visualizing the data
Can you write Python code for it?

no2.plot(kind='box', ylim=[0,250])

no2['BASCH'].plot(kind='hist', bins=50)


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Recap: aggregating functions
When analyzing data, you often calculate summary statistics (aggregations like the mean, max, ...). As we have seen before, we can easily calculate such a statistic for a Series or column using one of the many available methods. For example:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Selecting by position with iloc works similar as indexing numpy arrays:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Starting from reading this dataset, to answering questions about this data in a few lines of code:
What is the age distribution of the passengers?
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
The default plot (when not specifying kind) is a line plot of all columns:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
The different indexing methods can also be used to assign data:
Can you write Python code for it?

df.loc['Braund, Mr. Owen Harris', 'Survived'] = 100

df


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
The string to specify the new time frequency: http://pandas.pydata.org/pandas-docs/dev/timeseries.html#offset-aliases
These strings can also be combined with numbers, eg '10D'.
Further exploring the data:
Can you write Python code for it?

no2.resample('M').mean().plot() # 'A'

# no2['2012'].resample('D').plot()

# %load snippets/02-pandas_introduction96.py


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
Unlike the NumPy array, though, this index can be something other than integers:
Can you write Python code for it?

df = df.set_index('Name')
df

age = df['Age']
age

age['Dooley, Mr. Patrick']


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
We can access series values via the index, just like for NumPy arrays:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
We will come back to these example, and build them up step by step.
2. Pandas: data analysis in python
For data-intensive work in Python the Pandas library has become essential.
What is pandas?

Pandas can be thought of as NumPy arrays with labels for rows and columns, and better support for heterogeneous data types, but it's also much, much more than that.
Pandas can also be thought of as R's data.frame in Python.
Powerful for working with missing data, working with time series data, for reading and writing your data, for reshaping, grouping, merging your data, ...

It's documentation: http://pandas.pydata.org/pandas-docs/stable/
 When do you need pandas?
When working with tabular or structured data (like R dataframe, SQL table, Excel spreadsheet, ...):

Import data
Clean up messy data
Explore data, gain insight into data
Process and prepare your data for analysis
Analyse your data (together with scikit-learn, statsmodels, ...)

ATTENTION!:

Pandas is great for working with heterogeneous and tabular 1D/2D data, but not all types of data fit in such structures!

When working with array data (e.g. images, numerical algorithms): just stick with numpy

When working with multidimensional labeled data (e.g. climate data): have a look at [xarray](http://xarray.pydata.org/en/stable/)

2. The pandas data structures: DataFrame and Series
A DataFrame is a tablular data structure (multi-dimensional object to hold labeled data) comprised of rows and columns, akin to a spreadsheet, database table, or R's data.frame object. You can think of it as multiple Series object which share the same index.

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
What is the difference in diurnal profile between weekdays and weekend?
Can you write Python code for it?

data['weekday'] = data.index.weekday
data['weekend'] = data['weekday'].isin([5, 6])
data_weekend = data.groupby(['weekend', data.index.hour])['BASCH'].mean().unstack(level=0)
data_weekend.plot()


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
When we ensure the DataFrame has a DatetimeIndex, time-series related functionality becomes available:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
You can access the underlying numpy array representation with the .values attribute:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
df[] provides some convenience shortcuts
For a DataFrame, basic indexing selects the columns.
Selecting a single column:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/02-pandas_introduction.ipynb".
The first task is:
to answering questions about this data in a few lines of code:
Does the air pollution show a decreasing trend over the years?
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
2. creating objects
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot(x, y, '-')


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:

 Remember:

You can do anything with matplotlib, but at a cost... [stackoverflow!!](http://stackoverflow.com/questions/tagged/matplotlib)

The preformatting of Pandas provides mostly enough flexibility for quick analysis and draft reporting. It is not for paper-proof figures or customization

 NOTE:

If you take the time to make you're perfect/spot-on/greatest-ever matplotlib-figure: Make it a **reusable function**! (see tomorrow!)

Let your hard work pay off, write your own custom functions!

 Remember:

`fig.savefig()` to save your Figure object!

Seaborn
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:

 Remember:

`lmplot` and `factorplot` are shortcuts for a more advanced `FacetGrid` functionality

If you want to dig deeper into this `FacetGrid`-based plotting, check the [online manual](http://seaborn.pydata.org/tutorial/axis_grids.html)!

Need more matplotlib/seaborn inspiration?
For more in-depth material:
* http://www.labri.fr/perso/nrougier/teaching/matplotlib/
* notebooks in matplotlib section: http://nbviewer.jupyter.org/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/Index.ipynb#4.-Visualization-with-Matplotlib
* main reference: matplotlib homepage
* very nice blogpost about customizing figures with matplotlib: http://pbpython.com/effective-matplotlib.html

 Remember(!)

[matplotlib Gallery](http://matplotlib.org/gallery.html)

[seaborn gallery ](http://seaborn.pydata.org/examples/index.html)

Important resources to start from!

Alternatives for matplotlib
We only use matplotlib (or matplotlib-based plotting) in this workshop, and it is still the main plotting library for many scientists, but it is not the only existing plotting library.
A nice overview of the landscape of visualisation tools in python was recently given by Jake VanderPlas: (or matplotlib-based plotting): https://speakerdeck.com/jakevdp/pythons-visualization-landscape-pycon-2017
Bokeh (http://bokeh.pydata.org/en/latest/): interactive, web-based visualisation
Can you write Python code for it?

from bokeh.io import output_notebook
output_notebook()

from bokeh.plotting import figure, show
from bokeh.sampledata.iris import flowers

colormap = {'setosa': 'red', 'versicolor': 'green', 'virginica': 'blue'}
colors = [colormap[x] for x in flowers['species']]

p = figure(title = "Iris Morphology")
p.xaxis.axis_label = 'Petal Length'
p.yaxis.axis_label = 'Petal Width'

p.circle(flowers["petal_length"], flowers["petal_width"],
         color=colors, fill_alpha=0.2, size=10)
show(p)


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:

 Remember:

Adapting the output of a Seaborn `grid` of different axes can be done as well to adapt it with matplotlib

Who likes regressions?
Can you write Python code for it?

fig, ax = plt.subplots()
sns.regplot(x="Fare", y="Age", data=titanic, ax=ax, lowess=False)
# adding the small lines to indicate individual data points
sns.rugplot(titanic["Fare"].dropna(), axis='x',
            color="#6699cc", height=0.02, ax=ax)
sns.rugplot(titanic["Age"].dropna(), axis='y',
            color="#6699cc", height=0.02, ax=ax)


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:

 Remember:

Similar to Pandas handling above, we can set up a `figure` and `axes` and add the seaborn output to it; adapt it afterwards

Compare two variables (scatter-plot):
Can you write Python code for it?

g = sns.jointplot(x="Fare", y="Age",
                  data=titanic,
                  kind="scatter") #kde, hex

g = sns.jointplot(x="Fare", y="Age",
                  data=titanic,
                  kind="scatter") #kde, hex
# Adapt the properties with matplotlib by changing the available axes objects
g.ax_marg_x.set_ylabel("Frequency")
g.ax_joint.set_facecolor('0.1')
g.ax_marg_y.set_xlabel("Frequency")


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
Best of both worlds...
Can you write Python code for it?

aqdata.columns

fig, ax = plt.subplots() #prepare a matplotlib figure

aqdata.plot(ax=ax) # use pandas for the plotting

# Provide further adaptations with matplotlib:
ax.set_xlabel("")
ax.tick_params(labelsize=15, pad=8, which='both')
fig.suptitle('Air quality station time series', fontsize=15)

fig, (ax1, ax2) = plt.subplots(2, 1) #provide with matplotlib 2 axis

aqdata[["BASCH", "BONAP"]].plot(ax=ax1) # plot the two timeseries of the same location on the first plot
aqdata["PA18"].plot(ax=ax2) # plot the other station on the second plot

# further adapt with matplotlib
ax1.set_ylabel("BASCH")
ax2.set_ylabel("PA18")
ax2.legend()


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
Comparison 2: with subplots
Can you write Python code for it?

axs = aqdata.plot(subplots=True, sharex=True,
                  figsize=(16, 8), colormap='viridis', # Dark2
                  fontsize=15)


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
For more information on legend positioning, check this post on stackoverflow!
Another nice blogpost about customizing matplotlib figures: http://pbpython.com/effective-matplotlib.html
I do not like the style...
The power of the object-oriented way of working makes it possible to change everything. However, mostly we just want quickly a good-looking plot. Matplotlib provides a number of styles that can be used to quickly change a number of settings:
Can you write Python code for it?

plt.style.available

x = np.linspace(0, 10)

with plt.style.context('seaborn-muted'):  # 'ggplot', 'bmh', 'grayscale', 'seaborn-whitegrid'
    fig, ax = plt.subplots()
    ax.plot(x, np.sin(x) + x + np.random.randn(50))
    ax.plot(x, np.sin(x) + 0.5 * x + np.random.randn(50))
    ax.plot(x, np.sin(x) + 2 * x + np.random.randn(50))


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
Making this with matplotlib...
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(16, 6))
ax.plot(aqdata.index, aqdata["BASCH"],
        aqdata.index, aqdata["BONAP"],
        aqdata.index, aqdata["PA18"],
        aqdata.index, aqdata["VERS"])
ax.legend(["BASCH", "BONAP", "PA18", "VERS"])


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
Matplotlib
Matplotlib is a Python package used widely throughout the scientific Python community to produce high quality 2D publication graphics. It transparently supports a wide range of output formats including PNG (and other raster formats), PostScript/EPS, PDF and SVG and has interfaces for all of the major desktop GUI (graphical user interface) toolkits. It is a great package with lots of options.
However, matplotlib is...

The 800-pound gorilla — and like most 800-pound gorillas, this one should probably be avoided unless you genuinely need its power, e.g., to make a custom plot or produce a publication-ready graphic.
(As we’ll see, when it comes to statistical visualization, the preferred tack might be: “do as much as you easily can in your convenience layer of choice [nvdr e.g. directly from Pandas, or with seaborn], and then use matplotlib for the rest.”)

(quote used from this blogpost)
And that's we mostly did, just use the .plot function of Pandas. So, why do we learn matplotlib? Well, for the ...then use matplotlib for the rest.; at some point, somehow!
Matplotlib comes with a convenience sub-package called pyplot which, for consistency with the wider matplotlib community, should always be imported as plt:
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
Matplotlib's pyplot module makes the process of creating graphics easier by allowing us to skip some of the tedious Artist construction. For example, we did not need to manually create the Figure artist with plt.figure because it was implicit that we needed a figure when we created the Axes artist.
Under the hood matplotlib still had to create a Figure artist, its just we didn't need to capture it into a variable. We can access the created object with the "state" functions found in pyplot called gcf and gca.
- essential stuff - pyplot versus Object based
Some example data:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
Observe the following difference:
1. pyplot style: plt... (you will see this a lot for code online!)
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
On its own, drawing the figure artist is uninteresting and will result in an empty piece of paper (that's why we didn't see anything above).
By far the most useful artist in matplotlib is the "Axes" artist. The Axes artist represents the "data space" of a typical plot, a rectangular axes (the most common, but not always the case, e.g. polar plots) will have 2 (confusingly named) Axis artists with tick labels and tick marks.
There is no limit on the number of Axes artists which can exist on a Figure artist. Let's go ahead and create a figure with a single Axes artist, and show it using pyplot:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
Seaborn works well with Pandas & is built on top of Matplotlib
We will use the Titanic example again:
Can you write Python code for it?

titanic = pd.read_csv('data/titanic.csv')

titanic.head()


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
Section especially for R ggplot lovers
Regressions with factors/categories: lmplot
When you want to take into account a category as well to do regressions, use lmplot (which is a special case of Facetgrid):
Can you write Python code for it?

sns.lmplot(x="Fare", y="Age", hue="Sex",
           data=titanic)

sns.lmplot(x="Fare", y="Age", hue="Sex",
           col="Survived", data=titanic)


I am working on the file "Day_1_Scientific_Python/03-matplotib_seaborn.ipynb".
The first task is:
The pandas versus matplotlib
Comparison 1: single plot
Can you write Python code for it?

aqdata.plot(figsize=(16, 6)) # shift tab this!


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
4. Numerical integration - scipy.integrate
Given a function object, the most generic integration routine is scipy.integrate.quad().
Can you write Python code for it?

from scipy.integrate import quad
res, err = quad(np.sin, 0, np.pi / 2)
res


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:

EXERCISE - `scipy.integrate`:

We would be interested in the area under the Gaussian profile since it is related to what we want to quantify.

Using `scipy.integrate.simps`, compute the area under the Gaussian profile between $[\mu - 3 \sigma, \mu + 3 \sigma]$. Those parameters can be found as the results of the curve fitting previusly done. The indexes corresponding to the interval values can be computed using `find_nearest_index`.

You can do the same using the original data to see the difference od quantification.

5. Linear algebra - scipy.linalg
The scipy.linalg offers basic operation used in linear algebra such as inverse (scipy.linalg.inv), pseudo-inverse (scipy.linalg.pinv), determinant (scipy.linalg.det) as well as decompostion as standard decompisition as SVD, QR, or Cholesky among others.

`np.array` vs. `np.matrix`:

By default the multiplication between two `np.array` (i.e. `*` operator) do not lead to a matrix multiplication. You need to use `np.dot` to perform this operation.

Another possibility is to convert the `np.array` to `np.matrix` which perform this operation when using the operator `*`. The operations become more readable when there is a lot of algebric operations involved.

We illustrate this behaviour in the example below.

Let's declare two arrays of shape $3 \times 3$ and $3 \times 1$, respectively.
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:

EXERCISE - `scipy.interpolate`:

Interpolate each spectra values corresponding to the integral frequencies {401, 402, ..., 3999} using `scipy.interpolate.interp1d`.

Plot the spectra in function of the frequencies.

3. Optimization - scipy.optimize
Optimization is the problem of finding a numerical solution to a minimization or equality.
The scipy.optimize module provides useful algorithms for function minimization (scalar or multi-dimensional), curve fitting and root finding.
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:

EXERCISE - `scipy.linalg`:

Solve the following system of linear equations using the normal equation.

$$
\left[\begin{array}{cc}
3x & 3y & -z \\
2x & -3y & 4z \\
-x & .5y & -z
\end{array}\right]
\left[\begin{array}{cc}
x_1 \\
x_2 \\
x_3
\end{array}\right] =
\left[\begin{array}{cc}
-1 \\
-2 \\
0
\end{array}\right]
$$

This problem can be seen as:
$$ A x = b $$

$x$ can be find such that:

$$ x = (A^{T} A)^{-1} A^{T} b $$

Find $x$ using the above equation

EXERCISE:

Solve the following system of linear equations using SVD.

The above problem can also be solved using an SVD decomposition such that:

$$ x = V S^{-1} (U^{T} b) $$

where $U$, $S$, and $V^{T}$ can be found with `scipy.linalg.svd` such that:
`U, S, Vh = svd(A)`

6. Statistics - scipy.stats and statsmodel
scipy.stats
scipy.stats contains mainly helper of most common continuous and discrete distribution.
In addition, this module contain statistical functions to perform statistical tests for instance.
Can you write Python code for it?

import pandas as pd
data = pd.read_csv('data/brain_size.csv', sep=';', na_values=".")
data.head()


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:

EXERCISE:

Test the difference between weights in males and females. You can fill the missing data using `pandas.fillna()` and using the mean weight of the population.

Use non parametric statistics to test the difference between VIQ in males and females (refer to `scipy.stats.mannwhitneyu`).

statsmodels
Given two set of observations, x and y, we want to test the hypothesis that y is a linear function of x. In other terms:
$$
    y = x \times coef + intercept + e
$$
where e is observation noise. We will use the statsmodels module to:

Fit a linear model. We will use the simplest strategy, ordinary least squares (OLS).
Test that coef is non zero.
Can you write Python code for it?

x = np.linspace(-5, 5, 20)
np.random.seed(1)
# normal distributed noise
y = -5 + 3 * x + 4 * np.random.normal(size=x.shape)
# Create a data frame containing all the relevant variables
data = pd.DataFrame({'x': x, 'y': y})


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
Curve fitting
Suppose we have data sampled from $f$ with some noise:
Can you write Python code for it?

xdata = np.linspace(-10, 10, num=100)
ydata = f(xdata) + np.random.normal(0, 2, xdata.shape)


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
Finding the minimum of a scalar function
Let’s define the following function:
Can you write Python code for it?

def f(x):
    return x ** 2 + 10 * np.sin(x)


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
Finding the roots of a scalar function
To find a root, i.e. a point where $f(x) = 0$, of the function f above we can use for example scipy.optimize.fsolve():
Can you write Python code for it?

root = optimize.fsolve(f, 1)  # our initial guess is 1
root


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
However, by converting $A$ and $b$ to matrices (i.e., np.matrix), it is possible to use the * operator directly.
Can you write Python code for it?

A = np.matrix(A)
b = np.matrix(b)

A * b


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
If only fixed sample are given, the trapeze method (scipy.integrate.trapz()) or Simpson's integration rule scipy.integrate.simps()) can be used.
Can you write Python code for it?

x = np.linspace(0, np.pi / 2, num=200)
y = np.sin(x)

from scipy.integrate import simps
res = simps(y, x)
res


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
Intercept: We can remove the intercept using - 1 in the formula, or force the use of an intercept using + 1.
Let's see another example: is VIQ can be predicted using Gender.
Can you write Python code for it?

from statsmodels.formula.api import ols
data = pd.read_csv('data/brain_size.csv', sep=';', na_values=".")
model = ols("VIQ ~ Gender + 1", data).fit()
print(model.summary())


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
Let's see the difference by plotting the results.
Can you write Python code for it?

plt.plot(measured_time, measures, 'or', label='Measures')
plt.plot(computed_time, linear_results, label='Linear interpolation')
plt.plot(computed_time, cubic_results, label='Cubic interpolation')
plt.legend()
plt.xlabel('Time')
plt.ylabel('Amplitude')
plt.show()


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
The SciPy library is one of the core packages that make up the SciPy stack. It provides many user-friendly and efficient numerical routines such as routines for numerical integration and optimization.
1. File input/output - scipy.io
Scipy provides an io module to help load some data type. We can easily read MATLAB .mat files using io.loadmat and io.savemat.
Can you write Python code for it?

from scipy.io import loadmat, savemat
a = np.ones((3, 3))
savemat('file.mat', {'a': a}) # savemat expects a dictionary
data = loadmat('file.mat', struct_as_record=True)
data['a']


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
The scipy.interpolate.interp1d class can build a linear interpolation function:
Can you write Python code for it?

from scipy.interpolate import interp1d
linear_interp = interp1d(measured_time, measures)


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
Then we specify an OLS model and fit it:
Can you write Python code for it?

from statsmodels.formula.api import ols
model = ols("y ~ x + 1", data).fit()


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
This function has a global minimum around -1.3 and a local minimum around 3.8.
The general and efficient way to find a minimum for this function is to conduct a gradient descent starting from a given initial point. The BFGS algorithm is a good way of doing this:
Can you write Python code for it?

res = optimize.minimize(f, 0, method='L-BFGS-B')
res


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
To see if this difference is significant, we can use scipy.stats.ttest_ind().
Can you write Python code for it?

from scipy.stats import ttest_ind
female_viq = data[data['Gender'] == 'Female']['VIQ']
male_viq = data[data['Gender'] == 'Male']['VIQ']
ttest_ind(female_viq, male_viq)


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
Using the * operator does not lead to a matrix multiplication since the matrix returned is a $3 \times 3$ matrix. Instead, it multiply each column of $A$ by the vector $b$.
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
We can inspect the various statistics derived from the fit:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
With a p-value of $10^{-28}$ we can claim that the population mean for the IQ (VIQ measure) is not 0.
2-sample t-test
scipy.stats.ttest_ind() can compare two populations and check if the difference is significant or not. We can study if there is a difference of the VIQ between Male and Female.
Can you write Python code for it?

groupby_gender = data.groupby('Gender')
for gender, value in groupby_gender['VIQ']:
    print((gender, value.mean()))


I am working on the file "Day_1_Scientific_Python/04-scipy_statsmodels_introduction.ipynb".
The first task is:
You need to use the function np.dot to obtain the matrix multiplication.
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
1. Let's start with a showcase
Case 1: titanic survival data
Can you write Python code for it?

df = pd.read_csv("data/titanic.csv")

df.head()


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
3. Data import and export
A wide range of input/output formats are natively supported by pandas:

CSV, text
SQL database
Excel
HDF5
json
html
pickle
sas, stata
(parquet)
...
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
4. Exploration
Some useful methods:
head and tail
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
6. The group-by operation
Some 'theory': the groupby operation (split-apply-combine)
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
7. Working with time series data
Can you write Python code for it?

no2 = pd.read_csv('data/20000101_20161231-NO2.csv', sep=';', skiprows=[1], na_values=['n/d'], index_col=0, parse_dates=True)


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Based on the titanic data set, how many passengers older than 70 were on the Titanic?

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Calculate the average survival ratio for all passengers.

Can you write Python code for it?

# df['Survived'].sum() / len(df['Survived'])
df['Survived'].mean()


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Calculate this survival ratio for all passengers younger that 25 (remember: filtering/boolean indexing).

Can you write Python code for it?

df25 = df[df['Age'] <= 25]
df25['Survived'].sum() / len(df25['Survived'])


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Make a bar plot to visualize the average Fare payed by people depending on their age. The age column is devided is separate classes using the `pd.cut` function as provided below.

Can you write Python code for it?

df['AgeClass'] = pd.cut(df['Age'], bins=np.arange(0,90,10))

df.groupby('AgeClass')['Fare'].mean().plot(kind='bar', rot=0)


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Or how does it differ between the different classes? Make a bar plot visualizing the survival ratio for the 3 classes.

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

Using groupby(), calculate the average age for each sex.

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:

EXERCISE:

What is the difference in the survival ratio between the sexes?

Can you write Python code for it?

# df.groupby('Sex')[['Survived']].aggregate(lambda x: x.sum() / len(x))
df.groupby('Sex')[['Survived']].mean()


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:

EXERCISE: The evolution of the yearly averages with, and the overall mean of all stations

Use `resample` and `plot` to plot the yearly averages for the different stations.

The overall mean of all stations can be calculated by taking the mean of the different columns (`.mean(axis=1)`).

Can you write Python code for it?

no2.resample('A').mean().plot()
no2.mean(axis=1).resample('A').mean().plot(color='k', linestyle='--', linewidth=4)


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:

EXERCISE: The typical diurnal profile for the different stations

Similar as for the month, you can now group by the hour of the day.

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:

EXERCISE: What is the difference in the typical diurnal profile between week and weekend days for the 'BASCH' station.

Add a column 'weekday' defining the different days in the week.

Add a column 'weekend' defining if a days is in the weekend (i.e. days 5 and 6) or not (True/False).

You can groupby on multiple items at the same time. In this case you would need to group by both weekend/weekday and hour of the day.

Add a column indicating the weekday:
Can you write Python code for it?

no2.index.weekday?

no2['weekday'] = no2.index.weekday


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
A nice feature is "partial string" indexing, so you don't need to provide the full datetime string.
E.g. all data of January up to March 2012:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
A range of methods:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Above I take the mean, but as with groupby I can also specify other methods:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Also a DataFrame has a values attribute, but attention: when you have heterogeneous data, all values will be upcasted:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
An overview of that information can be given with the info() method:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Apart from importing your data from an external source (text file, excel, database, ..), one of the most common ways of creating a dataframe is from a dictionary of arrays or lists.
Note that in the IPython notebook, the dataframe will display in a rich HTML view:
Can you write Python code for it?

data = {'country': ['Belgium', 'France', 'Germany', 'Netherlands', 'United Kingdom'],
        'population': [11.3, 64.3, 81.3, 16.9, 64.9],
        'area': [30510, 671308, 357050, 41526, 244820],
        'capital': ['Brussels', 'Paris', 'Berlin', 'Amsterdam', 'London']}
df_countries = pd.DataFrame(data)
df_countries


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Application of the groupby concept on the titanic data
We go back to the titanic passengers survival data:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Attributes of a Series: index and values
The Series has also an index and values attribute, but no columns
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Boolean indexing (filtering)
Often, you want to select rows based on a certain condition. This can be done with 'boolean indexing' (like a where clause in SQL) and comparable to numpy.
The indexer (or boolean mask) should be 1-dimensional and the same length as the thing being indexed.
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Fancy indexing, like indexing with a list or boolean indexing:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Getting some basic summary statistics about the data with describe:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
How does the survival rate of the passengers differ between sexes?
Can you write Python code for it?

df.groupby('Sex')[['Survived']].aggregate(lambda x: x.sum() / len(x))


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
However, in many cases your data has certain groups in it, and in that case, you may want to calculate this statistic for each of the groups.
For example, in the above dataframe df, there is a column 'key' which has three possible values: 'A', 'B' and 'C'. When we want to calculate the sum for each of those groups, we could do the following:
Can you write Python code for it?

for key in ['A', 'B', 'C']:
    print(key, df[df['key'] == key]['data'].sum())


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Luckily, if we have a well formed csv file, we don't need many of those arguments:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Or how does it differ between the different classes?
Can you write Python code for it?

df.groupby('Pclass')['Survived'].aggregate(lambda x: x.sum() / len(x)).plot(kind='bar')


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Or we can use some more advanced time series features -> see further in this notebook!
5. Selecting and filtering data

ATTENTION!:

One of pandas' basic features is the labeling of rows and columns, but this makes indexing also a bit more complex compared to numpy.

 We now have to distuinguish between:

selection by **label**

selection by **position**

Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Quickly visualizing the data
Can you write Python code for it?

no2.plot(kind='box', ylim=[0,250])

no2['BASCH'].plot(kind='hist', bins=50)


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Recap: aggregating functions
When analyzing data, you often calculate summary statistics (aggregations like the mean, max, ...). As we have seen before, we can easily calculate such a statistic for a Series or column using one of the many available methods. For example:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Selecting by position with iloc works similar as indexing numpy arrays:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Starting from reading this dataset, to answering questions about this data in a few lines of code:
What is the age distribution of the passengers?
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Systematic indexing with loc and iloc
When using [] like above, you can only select from one axis at once (rows or columns, not both). For more advanced indexing, you have some extra attributes:

loc: selection by label
iloc: selection by position
Can you write Python code for it?

df = df.set_index('Name')

df.loc['Bonnell, Miss. Elizabeth', 'Fare']

df.loc['Bonnell, Miss. Elizabeth':'Andersson, Mr. Anders Johan', :]


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
The default plot (when not specifying kind) is a line plot of all columns:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
The different indexing methods can also be used to assign data:
Can you write Python code for it?

df.loc['Braund, Mr. Owen Harris', 'Survived'] = 100

df


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
The power of pandas: resample
A very powerfull method is resample: converting the frequency of the time series (e.g. from hourly to daily data).
Remember the air quality data:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
The string to specify the new time frequency: http://pandas.pydata.org/pandas-docs/dev/timeseries.html#offset-aliases
These strings can also be combined with numbers, eg '10D'.
Further exploring the data:
Can you write Python code for it?

no2.resample('M').mean().plot() # 'A'

# no2['2012'].resample('D').plot()

no2.loc['2009':, 'VERS'].resample('M').agg(['mean', 'median']).plot()


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
This becomes very verbose when having multiple groups. You could make the above a bit easier by looping over the different values, but still, it is not very convenient to work with.
What we did above, applying a function on different groups, is a "groupby operation", and pandas provides some convenient functionality for this.
Groupby: applying functions per group
The "group by" concept: we want to apply the same function on subsets of your dataframe, based on some key to split the dataframe in subsets
This operation is also referred to as the "split-apply-combine" operation, involving the following steps:

Splitting the data into groups based on some criteria
Applying a function to each group independently
Combining the results into a data structure

<img src="../img/splitApplyCombine.png">
Similar to SQL GROUP BY
Instead of doing the manual filtering as above
df[df['key'] == "A"].sum()
df[df['key'] == "B"].sum()
...

pandas provides the groupby method to do exactly this:
Can you write Python code for it?

df.groupby('key').sum()

df.groupby('key').aggregate(np.sum)  # 'sum'


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
This does not say too much ..
We can select part of the data (eg the latest 500 data points):
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
To check the data types of the different columns:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
Very powerful csv reader:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
We can access series values via the index, just like for NumPy arrays:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
but with the power of numpy arrays. Many things you can do with numpy arrays, can also be applied on DataFrames / Series.
Eg element-wise operations:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
df[] provides some convenience shortcuts
For a DataFrame, basic indexing selects the columns.
Selecting a single column:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
info()
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__02-pandas_introduction.ipynb".
The first task is:
or multiple columns:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
1-sample t-test
scipy.stats.ttest_1samp() tests if the population mean of data is likely to be equal to a given value. Let see if the VIQ of our population is equal to 0.
Can you write Python code for it?

from scipy.stats import ttest_1samp

ttest_1samp(data['VIQ'], 0)


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
2. Signal interpolation - scipy.interpolate
The scipy.interpolate is useful for fitting a function from experimental data and thus evaluating points where no measure exists. By imagining experimental data close to a sine function:
Can you write Python code for it?

measured_time = np.linspace(0, 1, 10)
noise = (np.random.random(10)*2 - 1) * 1e-1
measures = np.sin(2 * np.pi * measured_time) + noise


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
4. Numerical integration - scipy.integrate
Given a function object, the most generic integration routine is scipy.integrate.quad().
Can you write Python code for it?

from scipy.integrate import quad
res, err = quad(np.sin, 0, np.pi / 2)
res


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
6. Statistics - scipy.stats and statsmodel
scipy.stats
scipy.stats contains mainly helper of most common continuous and discrete distribution.
In addition, this module contain statistical functions to perform statistical tests for instance.
Can you write Python code for it?

import pandas as pd
data = pd.read_csv('data/brain_size.csv', sep=';', na_values=".")
data.head()


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:

EXERCISE - `scipy.interpolate`:

Interpolate each spectra values corresponding to the integral frequencies {401, 402, ..., 3999} using `scipy.interpolate.interp1d`.

Plot the spectra in function of the frequencies.

Can you write Python code for it?

from scipy.interpolate import interp1d

frequency_interp = np.arange(401, 3999, 1)

spectra_interp = []
for spectrum in spectra:
    f_spectra = interp1d(frequency, spectrum)
    spectra_interp.append(f_spectra(frequency_interp))

# convert to a numpy array
spectra_interp = np.array(spectra_interp)
spectra_interp

plt.plot(frequency_interp, spectra_interp.T)
plt.show()


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:

EXERCISE:

Run an OLS to check if Weight can be predicted using Gender and Height.

Can you write Python code for it?

data = data.fillna(data.mean())
model = ols("Weight ~ Gender + Height + 1", data).fit()
print(model.summary())


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:

EXERCISE:

Solve the following system of linear equations using SVD.

The above problem can also be solved using an SVD decomposition such that:

$$ x = V S^{-1} (U^{T} b) $$

where $U$, $S$, and $V^{T}$ can be found with `scipy.linalg.svd` such that:
`U, S, Vh = svd(A)`

Can you write Python code for it?

from scipy.linalg import svd

U, S, V = svd(A)

# convert into matrix to simplify the multiplication
U = np.matrix(U)
S = np.matrix(np.diag(S))
V = np.matrix(V)

x = V.T * inv(S) * (U.T * b)

print('The solution x is equal to:')
print(x)


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:

EXERCISE:

Test the difference between weights in males and females. You can fill the missing data using `pandas.fillna()` and using the mean weight of the population.

Use non parametric statistics to test the difference between VIQ in males and females (refer to `scipy.stats.mannwhitneyu`).

Can you write Python code for it?

groupby_gender = data.groupby('Gender')
for gender, value in groupby_gender['Weight']:
    print((gender, value.mean()))

from scipy.stats import ttest_ind
female_weight = data[data['Gender'] == 'Female']['Weight']
male_weight = data[data['Gender'] == 'Male']['Weight']
# fill the missing data with the mean
femaile_weight = female_weight.fillna(female_weight.mean())
male_weight = male_weight.fillna(male_weight.mean())
# compute the t-test
ttest_ind(female_weight, male_weight)

from scipy.stats import mannwhitneyu
mannwhitneyu(female_viq, male_viq)


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
A cubic interpolation can also be selected by providing the kind optional keyword argument:
Can you write Python code for it?

cubic_interp = interp1d(measured_time, measures, kind='cubic')
cubic_results = cubic_interp(computed_time)


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
A possible issue with this approach is that, if the function has local minima the algorithm may find these local minima instead of the global minimum depending on the initial point:
Can you write Python code for it?

res2 = optimize.minimize(f, 3, method='L-BFGS-B')
res2


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
Curve fitting
Suppose we have data sampled from $f$ with some noise:
Can you write Python code for it?

xdata = np.linspace(-10, 10, num=100)
ydata = f(xdata) + np.random.normal(0, 2, xdata.shape)


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
Finding the roots of a scalar function
To find a root, i.e. a point where $f(x) = 0$, of the function f above we can use for example scipy.optimize.fsolve():
Can you write Python code for it?

root = optimize.fsolve(f, 1)  # our initial guess is 1
root


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
If we don’t know the neighborhood of the global minimum to choose the initial point, we need to resort to costlier global optimization. To find the global minimum, we use scipy.optimize.basinhopping() (which combines a local optimizer with stochastic sampling of starting points for the local optimizer):
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
Note that only one root is found. Inspecting the plot of f reveals that there is a second root around -2.5. We find the exact value of it by adjusting our initial guess:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
Summary in a single plot
Can you write Python code for it?

x = np.arange(-10, 10, 0.1)
plt.plot(xdata, ydata)
# plot the local minima
plt.plot(res.x, f(res.x), 'or', label='minimum')
plt.plot(res2.x, f(res2.x), 'or')
# plot the roots
plt.plot(root, f(root), '^g', label='roots')
plt.plot(root2, f(root2), '^g')
# plot the curved fitted
plt.plot(x, f2(x, params[0], params[1]), '--', label='fitted')
plt.legend()
plt.show()


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
The SciPy library is one of the core packages that make up the SciPy stack. It provides many user-friendly and efficient numerical routines such as routines for numerical integration and optimization.
1. File input/output - scipy.io
Scipy provides an io module to help load some data type. We can easily read MATLAB .mat files using io.loadmat and io.savemat.
Can you write Python code for it?

from scipy.io import loadmat, savemat
a = np.ones((3, 3))
savemat('file.mat', {'a': a}) # savemat expects a dictionary
data = loadmat('file.mat', struct_as_record=True)
data['a']


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
The scipy.interpolate.interp1d class can build a linear interpolation function:
Can you write Python code for it?

from scipy.interpolate import interp1d
linear_interp = interp1d(measured_time, measures)


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
Then we can use scipy.optimize.curve_fit() to find $a$ and $b$:
Can you write Python code for it?

guess = [2, 2]
params, params_covariance = optimize.curve_fit(f2, xdata, ydata, guess)
params


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
Then we specify an OLS model and fit it:
Can you write Python code for it?

from statsmodels.formula.api import ols
model = ols("y ~ x + 1", data).fit()


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
This function has a global minimum around -1.3 and a local minimum around 3.8.
The general and efficient way to find a minimum for this function is to conduct a gradient descent starting from a given initial point. The BFGS algorithm is a good way of doing this:
Can you write Python code for it?

res = optimize.minimize(f, 0, method='L-BFGS-B')
res


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
Using the * operator does not lead to a matrix multiplication since the matrix returned is a $3 \times 3$ matrix. Instead, it multiply each column of $A$ by the vector $b$.
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
We can inspect the various statistics derived from the fit:
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
With a p-value of $10^{-28}$ we can claim that the population mean for the IQ (VIQ measure) is not 0.
2-sample t-test
scipy.stats.ttest_ind() can compare two populations and check if the difference is significant or not. We can study if there is a difference of the VIQ between Male and Female.
Can you write Python code for it?

groupby_gender = data.groupby('Gender')
for gender, value in groupby_gender['VIQ']:
    print((gender, value.mean()))


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
You need to use the function np.dot to obtain the matrix multiplication.
Can you write Python code for it?


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
and plot it:
Can you write Python code for it?

x = np.arange(-10, 10, 0.1)
plt.plot(x, f(x))
plt.show()


I am working on the file "Day_1_Scientific_Python/__solved__04-scipy_statsmodels_introduction.ipynb".
The first task is:
statsmodels
Given two set of observations, x and y, we want to test the hypothesis that y is a linear function of x. In other terms:
$$
    y = x \times coef + intercept + e
$$
where e is observation noise. We will use the statsmodels module to:

Fit a linear model. We will use the simplest strategy, ordinary least squares (OLS).
Test that coef is non zero.
Can you write Python code for it?

x = np.linspace(-5, 5, 20)
np.random.seed(1)
# normal distributed noise
y = -5 + 3 * x + 4 * np.random.normal(size=x.shape)
# Create a data frame containing all the relevant variables
data = pd.DataFrame({'x': x, 'y': y})


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.1-Machine-Learning-Intro.ipynb".
The first task is:
Again, this is an example of fitting a model to data, such that the model can make
generalizations about new data.  The model has been learned from the training
data, and can be used to predict the result of test data:
here, we might be given an x-value, and the model would
allow us to predict the y value.  Again, this might seem like a trivial problem,
but it is a basic example of a type of operation that is fundamental to
machine learning tasks.
Representation of Data in Scikit-learn
Machine learning is about creating models from data: for that reason, we'll start by
discussing how data can be represented in order to be understood by the computer.  Along
with this, we'll build on our matplotlib examples from the previous section and show some
examples of how to visualize data.
Most machine learning algorithms implemented in scikit-learn expect data to be stored in a
two-dimensional array or matrix.  The arrays can be
either numpy arrays, or in some cases scipy.sparse matrices.
The size of the array is expected to be [n_samples, n_features]

n_samples:   The number of samples: each sample is an item to process (e.g. classify).
  A sample can be a document, a picture, a sound, a video, an astronomical object,
  a row in database or CSV file,
  or whatever you can describe with a fixed set of quantitative traits.
n_features:  The number of features or distinct traits that can be used to describe each
  item in a quantitative manner.  Features are generally real-valued, but may be boolean or
  discrete-valued in some cases.

The number of features must be fixed in advance. However it can be very high dimensional
(e.g. millions of features) with most of them being zeros for a given sample. This is a case
where scipy.sparse matrices can be useful, in that they are
much more memory-efficient than numpy arrays.
A Simple Example: the Iris Dataset
As an example of a simple dataset, we're going to take a look at the
iris data stored by scikit-learn.
The data consists of measurements of three different species of...
Can you write Python code for it?

from IPython.core.display import Image, display
display(Image(filename='images/iris_setosa.jpg'))
print("Iris Setosa\n")

display(Image(filename='images/iris_versicolor.jpg'))
print("Iris Versicolor\n")

display(Image(filename='images/iris_virginica.jpg'))
print("Iris Virginica")


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.1-Machine-Learning-Intro.ipynb".
The first task is:
Quick Exercise:
Change x_index and y_index in the above script
and find a combination of two parameters
which maximally separate the three classes.
This exercise is a preview of dimensionality reduction, which we'll see later.
Other Available Data
They come in three flavors:

Packaged Data: these small datasets are packaged with the scikit-learn installation,
  and can be downloaded using the tools in sklearn.datasets.load_*
Downloadable Data: these larger datasets are available for download, and scikit-learn
  includes tools which streamline this process.  These tools can be found in
  sklearn.datasets.fetch_*
Generated Data: there are several datasets which are generated from models based on a
  random seed.  These are available in the sklearn.datasets.make_*

You can explore the available dataset loaders, fetchers, and generators using IPython's
tab-completion functionality.  After importing the datasets submodule from sklearn,
type
datasets.load_ + TAB

or
datasets.fetch_ + TAB

or
datasets.make_ + TAB

to see a list of available functions.
Can you write Python code for it?

from sklearn import datasets

# Type datasets.fetch_<TAB> or datasets.load_<TAB> in IPython to see all possibilities

datasets.make_

# datasets.load_


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.1-Machine-Learning-Intro.ipynb".
The first task is:
Quick Question:
If we want to design an algorithm to recognize iris species, what might the data be?
Remember: we need a 2D array of size [n_samples x n_features].

What would the n_samples refer to?

What might the n_features refer to?

Remember that there must be a fixed number of features for each sample, and feature
number i must be a similar kind of quantity for each sample.
Loading the Iris Data with Scikit-Learn
Scikit-learn has a very straightforward set of data on these iris species.  The data consist of
the following:

Features in the Iris dataset:

sepal length in cm

sepal width in cm
petal length in cm

petal width in cm

Target classes to predict:

Iris Setosa

Iris Versicolour
Iris Virginica

scikit-learn embeds a copy of the iris CSV file along with a helper function to load it into numpy arrays:
Can you write Python code for it?

from sklearn.datasets import load_iris
iris = load_iris()

iris.keys()

n_samples, n_features = iris.data.shape
print((n_samples, n_features))
print(iris.data[0])

print(iris.data.shape)
print(iris.target.shape)

print(iris.target)

print(iris.target_names)


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.1-Machine-Learning-Intro.ipynb".
The first task is:
This data is four dimensional, but we can visualize two of the dimensions
at a time using a simple scatter-plot:
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt

x_index = 1
y_index = 3

# this formatter will label the colorbar with the correct target names
formatter = plt.FuncFormatter(lambda i, *args: iris.target_names[int(i)])

plt.scatter(iris.data[:, x_index], iris.data[:, y_index],
            c=iris.target, cmap=plt.cm.get_cmap('RdYlBu', 3))
plt.colorbar(ticks=[0, 1, 2], format=formatter)
plt.clim(-0.5, 2.5)
plt.xlabel(iris.feature_names[x_index])
plt.ylabel(iris.feature_names[y_index]);


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.1-Machine-Learning-Intro.ipynb".
The first task is:
This may seem like a trivial task, but it is a simple version of a very important concept.
By drawing this separating line, we have learned a model which can generalize to new
data: if you were to drop another point onto the plane which is unlabeled, this algorithm
could now predict whether it's a blue or a red point.
If you'd like to see the source code used to generate this, you can either open the
code in the figures directory, or you can load the code using the %load magic command:
The next simple task we'll look at is a regression task: a simple best-fit line
to a set of data:
Can you write Python code for it?

from fig_code import plot_linear_regression
plot_linear_regression()


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
A more useful way to look at the results is to view the confusion matrix, or the matrix showing the frequency of inputs and outputs:
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
print(confusion_matrix(y, y_pred))


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
As above, we can plot a line of best fit:
Can you write Python code for it?

model = LinearRegression()
model.fit(X, y)

# Plot the data and the model prediction
X_fit = np.linspace(0, 1, 100)[:, np.newaxis]
y_fit = model.predict(X_fit)

plt.plot(X.squeeze(), y, 'o')
plt.plot(X_fit.squeeze(), y_fit);


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
Clustering: K-means
Clustering groups together observations that are homogeneous with respect to a given criterion, finding ''clusters'' in the data.
Note that these clusters will uncover relevent hidden structure of the data only if the criterion used highlights it.
Can you write Python code for it?

from sklearn.cluster import KMeans
k_means = KMeans(n_clusters=3, random_state=0) # Fixing the RNG in kmeans
k_means.fit(X)
y_pred = k_means.predict(X)

plt.scatter(X_reduced[:, 0], X_reduced[:, 1], c=y_pred,
           cmap='RdYlBu');


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
Estimated Model parameters: When data is fit with an estimator, parameters are estimated from the data at hand. All the estimated parameters are attributes of the estimator object ending by an underscore:
Can you write Python code for it?

x = np.arange(10)
y = 2 * x + 1

print(x)
print(y)

plt.plot(x, y, 'o');

# The input data for sklearn is 2D: (samples == 10 x features == 1)
X = x[:, np.newaxis]
print(X)
print(y)

# fit the model on our data
model.fit(X, y)

# underscore at the end indicates a fit parameter
print(model.coef_)
print(model.intercept_)

# residual error around fit
model.residues_

model.score(X, y)


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
Estimator parameters: All the parameters of an estimator can be set when it is instantiated, and have suitable default values:
Can you write Python code for it?

model = LinearRegression(normalize=True)
print(model.normalize)

print(model)

model2 = LinearRegression()

model?


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
Here the data is simply each pixel value within an 8x8 grid:
Can you write Python code for it?

# The images themselves
print(digits.images.shape)
print(digits.images[0])

# The data for use in our algorithms
print(digits.data.shape)
print(digits.data[0])

# The target label
print(digits.target)


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
Let's use a simple logistic regression which (despite its confusing name) is a classification algorithm:
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
clf = LogisticRegression(penalty='l2')
clf.fit(Xtrain, ytrain)
ypred = clf.predict(Xtest)


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
Original source on the scikit-learn website
Quick Application: Optical Character Recognition
To demonstrate the above principles on a more interesting problem, let's consider OCR (Optical Character Recognition) – that is, recognizing hand-written digits.
In the wild, this problem involves both locating and identifying characters in an image. Here we'll take a shortcut and use scikit-learn's set of pre-formatted digits, which is built-in to the library.
Loading and visualizing the digits data
We'll use scikit-learn's data access interface and take a look at this data:
Can you write Python code for it?

from sklearn import datasets
digits = datasets.load_digits()
digits.images.shape


I am working on the file "Day_1_Scientific_Python/scikit-learn/02.2-Basic-Principles.ipynb".
The first task is:
Scikit-learn also has some more sophisticated models, which can respond to finer features in the data:
Can you write Python code for it?

# Fit a Random Forest
from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(n_estimators=10, max_depth=5)
model.fit(X, y)

# Plot the data and the model prediction
X_fit = np.linspace(0, 1, 100)[:, np.newaxis]
y_fit = model.predict(X_fit)

plt.plot(X.squeeze(), y, 'o')
plt.plot(X_fit.squeeze(), y_fit);