I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
12. A concluding demonstration

In this last section, I want to pose a challenge for us to solve.
I want you to create a function which will return the day of the week for a given date input. For example, this function should return THURSDAY for an input of 14-09-2017 (in DD-MM-YYYY) format. The function must be able to accept any date in the past or the future. It must retain its validity even when the date in the past century. Your function signature can be the following:
def weekday_from_date(day=1, month=1, year=2017):
    <code>
    return <weekday as an int or str>

When returning and int to represent a weekday, we use 1=Monday, 2=Tuesday, ...,6=Saturday, 0=Sunday.
I have actually prepared most of the coding already. You will need to just code in one small section to complete this
assignment.
How this function works
In order to determine the day of the week from a given date, the most straightforward way is to count the number of days starting from today to the target date. This is complicated only by the fact that months can have 28-31 days.
We first have to determine whether the target date is in the future or past. This tells us whether to add or subtract the day difference to the current week day.
Next of course is to determine the day difference between target date and current date.
An example will serve to illustrate the idea: Days between 15-09-2017 and 18-10-2020 = days between 15-09-2017 to 15-09-2020 + days between 15-09-2020 to 15-10-2020 + days between 15-10-2020 to 18-10-2020.
The correct day of week in terms of its integer code is just the remainder of current day of week $\pm$ day difference divided by 7 since there are seven days in a week.
The assignment
You will need to fill in the empty section for the function month_diff which is meant to calculate the number of days between two dates which differs only by month, e.g. it calculates the number of days between 1st April 2017 to 1st September 2017 and not 4th April 2017 to 7th May 2017.
What code will achieve the...
Can you write Python code for it?

from datetime import datetime

DAY_OF_WEEK = {1: "MONDAY", 2: "TUESDAY", 3:"WEDNESDAY", 4:"THURSDAY", 5:"FRIDAY", 6:"SATURDAY", 0:"SUNDAY"}

def todays_date():
    t0 = datetime.today()
    return t0.isoweekday(), t0.day, t0.month, t0.year

# Returns day difference if target date is within same month and year
def day_diff(start_date, end_date):
    return (end_date[0] - start_date[0])

# Returns day difference if target date may be in differing months but within same year.
# Remember to account for leap years!
def month_diff(start_date, end_date):
    start_month, end_month, end_year = start_date[1], end_date[1], end_date[2]
    total_days = 0

    for m in range(min(start_month, end_month), max(start_month, end_month)):

        # Enter your answer here

        # End of answer

    # It is quite possible that start_month exceeds end_month. In this case,
    # we are actually counting days "backwards"! We then have to actually return
    # the negative value so that this number of days is subtracted from the total.
    if start_month < end_month:
        return total_days
    else:
        return -1*total_days

# Returns day difference across different years
def year_diff(start_date, end_date):
    start_year, end_year = start_date[2], end_date[2]
    total_days = 0

    # Adjusting for the fact that in a leap year, the extra day occurs on the last day of Feb.
    leap_year_adj = 0
    if end_date[1] >= 3 and end_date[2]%4==0:
        leap_year_adj += 1
    if start_date[1] >= 3 and start_date[2]%4==0:
        leap_year_adj += -1

    for y in range(start_year, end_year):
        if y%4==0:
            total_days += 366
        else:
            total_days += 365
    return total_days + leap_year_adj

# Returns day of week for given date
def weekday_from_date(day, month, year):
    curr_date = todays_date()

    # Checking whether the target_date is in the future (relative to the current date)
    # or not
    conds =...


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
9. Python functions

Sometimes, a portion of code is reused over and over again in the entire script. To prevent repetitive coding, we are able to define our own custom defined functions using the def keyword. When invoked, functions will instruct the computer to perform a set list of instructions, possibly returning an output at the end. Functions can also be pre-defined and saved in another file (with extension .py) so that it can be used for another project.
You will have encountered many pre-defined functions in Python up to now. Functions like print, len, etc... But there are many other pre-built functions in Python which are extremely useful and makes code more transparent and readable.
We will also encounter what is known as anonymous functions which are defined using the lambda keyword. These are short functions and can only be defined in one line of code. While technically def does what lambda does, nevertheless lambda allows for less pendantic and more natural style of coding. You will encounter anonymous functions alot when working with pandas especially when cleaning up data frames.
Finally we introduce what is perhaps one of the more important concepts in Python programming: List comprehensions. These are syntatical shortcuts for the for loop which translate not only to better code style, but faster scripts.
9.1 Learning objectives
The objectives of this unit are:

To use the def keyword to defined functions.

Recall the terms  arguments, keyword arguments, the signature of a function.

To use lambda to define anonymous functions.

To use build in  functions: zip and enumerate.

To learn how to refactor for loops into list comprehension statements.

9.2 Let's build our own functions
All functions are defined using the def keyword. In general, the format of function definition looks like this:
def my_function_name(arg_1, arg_2, ..., arg_n) :

    code

    return <something, or None>

First, we tell Python that we are going to define a function by typing out def. Then we...
Can you write Python code for it?

# Our first function

def my_first_function():
    pass


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
9.3.1 Function scope
But isn't the variable prod defined already when we defined the function special_product? This happens because the variable prod is only available in the scope of the function. The global environment is another scope. In general, variables from one scope are not accessible in another scope with exeptions given by scoping rules (which are programming language dependant). For this course, it suffices to know that variables defined in the function scope will NOT be accessible from the global scope.
(This can be overriden using the global keyword. But this is not encouraged.)
9.4 Function defaults
When we define functions, all variables we define in the signature must be assigned values. We cannot leave any out.
Can you write Python code for it?


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
9.5 Passing arguments to functions by keyword
The arguments to a function have names, just as variables have names. The names of arguments to a function are called keywords. We can pass arguments to functions by assigning values explicitly to keywords like so:
my_function(keyword_1 = value_1, keyword_2 = value_2,...)

This gives enormous flexibility in using Python interactively. Most functions given in the matplotlib and seaborn libraries have many arguments almost all of have them have default values. However, we often use a few of these keywords and it is quite a pain to remember the exact sequence of arguments in the function signature. Passing argments to keywords allows us to pass arguments in any order convenient to us.
Can you write Python code for it?

special_product(1,3) == special_product(3,1)

special_product(x=1, y=3) == special_product( y=3, x=1)


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
9.5.1 An application: A function to search for primes
To end this section, below is a function to determine whether a number is prime or not. We use this to refactor our prime listing code from the previous unit.
Can you write Python code for it?

import math

def is_prime(p):
    """
    This function determines if p is prime or not.

    Returns:
        bool, True if p is prime.
    """
    m = int(math.floor(math.sqrt(p)))
    for d in range(2, m+1):
        if p%d == 0:
            return False
    return True

for p in range(2, 101):
    if is_prime(p):
        print(p)


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
As with lists, tuples can also be indexed and sliced. However, once assigned, individual components of a tuple cannot be changed. For example, the following code will raise and error
Can you write Python code for it?


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
As you can see, a new column has been added with the calculated column named ratio_sul_to_alc. However, we had to define a function named ratio which we may or may not use again. We would like to achieve the same thing, but without populating the global namespace with unnecessary variables.
So let's do the same thing but with lambda expressions.
Can you write Python code for it?

(wine.assign(ratio_sul_to_alc=lambda df: df.sulphates/df.alcohol)
     .head(5))


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
Here's how this could be achieved. We first define the function that calculates the ratio and then proceed to create the new calculated column.
Can you write Python code for it?

def ratio(df):
    """ This function calculates the ratio of sulphates to alcohol content in the wine dataframe

    Returns
        Series, shape (n_samples, ) Array containing the ratio of sulphate to alcohol content for each sample
    """
    ratio_col = df.sulphates/df.alcohol
    return ratio_col

(wine.assign(ratio_sul_to_alc=ratio)
     .head(5))


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
Imagine having to write such a snippet of code every time we need to do something with elements from two lists! As you can imagine, it can cause code to be bloated and distracts from the main logic of the program.
Here's how zip is typically used in a program.
Can you write Python code for it?

for tup in zip(my_colleagues, ages):
    name = tup[0]
    age = tup[1]
    print("%s's age is %d" % (name, age))


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
Notice that I did not need to put a return to indicate which output to pass to the global environment. This is because lambda expressions are meant to be written in one line, hence it is understood that that one line of code is the output.
10.1 Use cases
Lambda expressions are also known as anonymous functions because we rarely assign lambda expressions to variables. Instead, they are passed directly to keywords or as arguments to most pandas functions or methods. Here is an example to how this is used in a pandas dataframe.
In what follows, we intend to calculate the ratio of sulphates to alchohol content for each sample (row) and assign it as a new column to the data frame. The data frame is displayed below and has been assigned to variable named wine.
Can you write Python code for it?

import pandas as pd # Importing the pandas library

wine = pd.read_csv("winequality-red.csv", sep=';')

wine.sample(5)


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
Notice that printer is of class function but is given a name <lambda>. However, we can call printer just as we called my_first_function, by passing arguments to it.
Can you write Python code for it?


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
Notice that the entire script needed about 1.8 ms to execute. This isn't exactly a short amount of time as far as computers go. Just imagine that we have to do this for 10 times in a row!
Fortunately Python implements what is known as list comprehension which is a way of writing for loops in a more compact and abstracted way. If we think about a for loop to create a list element by element, the code will look something like this:
for i in iterable:

    do code and return result_i
    append(result_i) to list[i]

Python list comprehension provides an alternative syntax which does the exact same thing but faster. The syntax is:
[do code for i in iterable]

The code is enclosed in [ ] because we are using list comprehension to create a list using a set instruction for each item in the iterable (think of iterables as a list). If one is familiar with set notation from calculus courses, list comprehension is syntatically similiar to the following $${\, f(x_n) \mid x_n \in A}$$ where $f$ is some function meant to be evaluated element-wise on each element in a set $A$.
Now let's refactor the for loop above using list comprehension and time the script.
Can you write Python code for it?

%%timeit

serial_numbers = ['s'+str(i) for i in range(5000)]


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
Notice that they give the same answer. We will learn how to do this in detail in the next unit. For now, the purpose of this example is to illustrate how lambda expressions are a great help in simplifying and making code more compact and readable.
11. Built in functions

Python has some built in functions for coding purposes. While there are quite a few of them, the following two will be used quite often in handling dataframes and visualization. These are

zip is a utility function that produces tuples from two lists of equal length.
enumerate also creates a tuple in the form $ (i, item_i)$ for $0\leq i < $ len(items).

We will also cover the concept of list comprehension in this section as it is an important programming concept and syntax in Python.
11.1 zip
To understand what zip does, we need to describe a rather simple data structure called tuple. A tuple is like a list, with the difference being that its elements are immutable. Tuples are created by enclosing a list of objects seperated by commas within two round braces ()
Can you write Python code for it?


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
Passing arguments in correct sequence matters. Python will pass values to arguments according to the sequence as it was declared in the signature.
Can you write Python code for it?

# x = 1, and y =3
print(special_product(1,3))

# x =3 and y = 1
print(special_product(3,1))


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
Think of tuples as lists which you wish to protect from changing by accidental assignment. Another way of thinking about tuples are also as constant lists, or as "coordinates" in $\mathbb{R}^n$.
Given two lists with items $$ x_1, \ldots, x_n$$ and $$ y_1, \ldots, y_n$$ zip produces a new list of tuples in the following form $$ (x_1, y_1), \ldots, (x_n, y_n)$$
To understand how zip works, let's try to replicate its function using a for loop.
Can you write Python code for it?

zipped = list() # This creates and empty list

my_colleagues = ['Andy', 'Lisa', 'Dayton']
ages = [29, 24, 50]

for i in range(0,3):
    zipped.append((my_colleagues[i], ages[i]))

print(zipped)


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
To refactor this into a list comprehension statement, we first identify the code that is being looped over. That is
mk_list(month[0:3].upper())

However, this composite statement can be broken down into steps:

month[0:3] just extracts the first 3 letters from month

Calling the .upper() method on the string of 3 letters captilizes them.

Calling mk_list is essentially the task of appending the result of the previous two steps to the list short_name.

In list comprehension, the last step is taken care of. Thus, the essential part of the code is month[0:3].upper(). Now we identify the iterable: This is simply the list months (note plural).
What is the variable to indicate the particular month as we iterate over the lists of months? This is simply denoted by the name month (note singular). There is nothing particularly special about using month to name each object in the list months. I could have easily used mon as well. In that case, the essential part of the code which is being looped should be written mon[0:3].upper(). With that clarified, the list comprehension statement is
Can you write Python code for it?

short_name = [month[0:3].upper() for month in months]
print(short_name)


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
What  happened is that the function special_product performs the said operation on inputs 1 and 3. It then outputs the answer, in this case 7. We assign the output 7 to a variable named answer and print it.
Note that we do not need to explicit declare a variable to "capture" the answer. The following works too.
Can you write Python code for it?


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
What will happen if we try to display the variable prod directly?
Can you write Python code for it?


I am working on the file "Day 1 - Unit 1.3.ipynb".
The first task is:
my_first_function will print the string "Hello world!" whenever it is called. Calling a function basically means instructing Python to run the code contained in the function. Notice that after defining a function and running the cell, there is no output. But that doesn't mean nothing has happened. In fact, Python has populated the global namespace with a new name, my_first_function and is ready to do what ever has been coded into this function when it is called.
Can you write Python code for it?


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
.loc is labelled based selection.
Can you write Python code for it?

# Select value based on index label
s0.loc['b'] # return same asnwer as s0.iloc[1] or s0['b']

# Select a list of values based on labels
s0.loc[['a','c','e']] # return same answer as s0.iloc[[0,2,4]] or s0[['a','c','e']]


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
2.1.0 Series Structure
Series is a one-dimensional array capable of holding any data type, like int, str, float,....
It is similar to list in Python, but each element is lablled with an index.
Can you write Python code for it?

# Create a series from a list
s = pd.Series([2,3,5,7,11])
# Display the series
s

# Get the series values
s.values

# Get the series index
s.index

# Name the index
s0 = pd.Series([2,3,5,7,11], index=['a','b','c','d','e'])
s0

# Size/Length of series
s.size # same answer as len(s)

# Sort values
s.sort_values(ascending=False)


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
2.1.1 Basic Operations
Series allows us to use "vectorised operations". This means that arithmetic operations can be performed without the use of a for loop to iterate through elements of the Series.
Can you write Python code for it?


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
2.1.2 Indexing and Selecting Data using .iloc and .loc
How do we access specific elements in the series? There are two ways, and integer positionals method using the iloc method and a loc method which selects elements by their indexing label.
.iloc is integer location based selection (from 0, 1, 2 to length-1 of the axis)
Can you write Python code for it?

# Select the first value
s.iloc[0]

# Select the last value
s.iloc[-1]

# Select a contiguous subset of values
s.iloc[1:4]

# Select the first 3 values
s.iloc[:3]


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
2.1.3 Conditional Selection
How do we  select elements of a series based on some criteria. For example, select all elements in s which are less than 10.
Can you write Python code for it?

# Which values are < 10?
s < 10

# Select values based on boolean mask
s[s<10]

# How many terms are <10?
len(s[s<10])

# Select values which are 2 < x < 10.
s[(s>2) & (s<10)]

# Note that the following is not valid

s[2<s<10]


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
2.1.4 Missing Values
NaN (Not a Number) is the standard missing marker used in Pandas.
Can you write Python code for it?

# Create a new series
s1 = pd.Series([13,np.nan,19])
# Append new series
s2 = s.append(s1)
# observe the indices
s2

# Append new series, ignoring the index
s2 = s.append(s1,ignore_index=True)
s2

# Check for missing value
s2.isnull()

# How many missing values are there?
s2.isnull().sum()

# Drop missing values
s2.dropna()

# Replace missing values
s2.fillna(17)

# Replace a value by another value
# Eg: Replace 2 by 'even prime'
s2.replace(2,'even prime')


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
Q2: How many students are there in this class?
Can you write Python code for it?


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
Q3(b): What should you replace 'TX' with? Then, answer Q3(a) again.
Can you write Python code for it?

# Answer should return a number
t1 = t.replace('TX',np.nan)
t1.mean()


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
Q4: Who scored the highest mark?
Can you write Python code for it?

# Answer should return a string (name)
t1.idxmax()


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
Q5: Who failed the test (<50)?
Can you write Python code for it?

# Answer should return a list of names
t1[t1 < 50].index


I am working on the file "Day 1 - Unit 2.1 Series.ipynb".
The first task is:
Q6: What is the percentage of B+ and above (>=75 / number of students who took MST)? Leave your answer correct to 2 d.p.
Can you write Python code for it?

# Answer should return a number correct to 2 d.p.
x = t1[t1>=75]
round(100*len(x)/18,2)


I am working on the file "Day 1 - Unit 2.2 DataFrame.ipynb".
The first task is:
2.2.3 Working with rows
Can you write Python code for it?

# Select multiple rows
df[2:4]

# Select the last row
df[-1:]

# Select all but last row
df[:-1]

# Select all even rows
df[::2]

# Select by .iloc
df.iloc[0:2,1:3]

# Select by .loc
df.loc[0:1,['Pop','Area']]


I am working on the file "Day 1 - Unit 2.2 DataFrame.ipynb".
The first task is:
2.2.5 Case Study: Olympic Games
We can import data from a csv file by using pd.read_csv.
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html
We can import data from an Excel file by using pd.read_excel.
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_excel.html
Source of data: https://en.wikipedia.org/wiki/All-time_Olympic_Games_medal_table
Can you write Python code for it?

# Import data from Excel file
og = pd.read_excel('OlympicGames.xlsx')
# Display first 5 rows of dataframe
og.head()


I am working on the file "Day 1 - Unit 2.2 DataFrame.ipynb".
The first task is:
Exercise: Which region has the highest density? Can you get the answer without sorting?
Can you write Python code for it?

# method 1:
df['Region'][df['Density'].idxmax()]

# method 2: Change the index
df1 = df.set_index('Region')
df1['Density'].idxmax()


I am working on the file "Day 1 - Unit 2.2 DataFrame.ipynb".
The first task is:
Q3: Create a new column for total number of Olympic medals.
Can you write Python code for it?

og['Total'] = og['Gold']+og['Silver']+og['Bronze']
og.head()


I am working on the file "Day 1 - Unit 2.2 DataFrame.ipynb".
The first task is:
Q4: Select the row where Country = Singapore.
Can you write Python code for it?


I am working on the file "Day 1 - Unit 2.2 DataFrame.ipynb".
The first task is:
Q5: Which country has won the highest number of Gold medals?
Can you write Python code for it?

#og[og['Gold'] == og['Gold'].max()]['Country']
og['Country'][og['Gold'].idxmax()]


I am working on the file "Day 1 - Unit 2.2 DataFrame.ipynb".
The first task is:
Q6: How many countries participated in at least 25 Olympic games (100 years)? Return an integer.
Can you write Python code for it?

len(og.query("Games >= 25").loc[:, "Country"])


I am working on the file "Day 1 - Unit 2.2 DataFrame.ipynb".
The first task is:
Q7: Which are the top 3 countries with highest total Olympic medals?
Challenge! Can you return a list of countries in one line of code?
Can you write Python code for it?


I am working on the file "Day 1 - Unit 2.2 DataFrame.ipynb".
The first task is:
Q8: Out of the countries which have not won any Gold medals, which country has won the highest number of medals?
Challenge! Can you return the name of the country in one line of code?
Can you write Python code for it?

og['Country'][og[og['Gold']==0]['Total'].idxmax()]
og.Country[og.query("Gold == 0").Total.idxmax()]


I am working on the file "Day 1 - Unit 2.3 Data Manipulations.ipynb".
The first task is:
2.3 Data Manipulations
Content:
 - 2.3.1 Groupby: split-apply-combine
 - 2.3.2 Merging dataframes
 - 2.3.3 Melting dataframes (wide-form to long-form)
 - 2.3.4 Exercises
Import libraries
Can you write Python code for it?


I am working on the file "Day 1 - Unit 2.3 Data Manipulations.ipynb".
The first task is:
2.3.2 Merging dataframes
Import data from url: http://files.grouplens.org/datasets/movielens/ml-100k/u.data
This is the data set of 100,000 ratings by 943 users on 1682 movies. Each user has rated at least 20 movies.
The data is randomly ordered. This is a tab separated list of user_id | movie_id | rating | timestamp.
Can you write Python code for it?

data = pd.read_csv('http://files.grouplens.org/datasets/movielens/ml-100k/u.data',sep='\t',header=None)
data.columns = ['user_id','movie_id','rating','timestamp']
data.tail()


I am working on the file "Day 1 - Unit 2.3 Data Manipulations.ipynb".
The first task is:
2.3.3 Wide-form vs Long-form Dataframe
Can you write Python code for it?

quiz = pd.read_excel('Quiz.xlsx')
quiz.head()
# This data frame is in wide format

# Melt the wide-form df to long-form
quizlong = pd.melt(quiz,id_vars=['ID'],value_vars=['Quiz1','Quiz2','Quiz3','Quiz4','Quiz5'],var_name='Quiz',value_name='Score')
quizlong.head()

# Shape of the long-form df
quizlong.shape

# Change long-form to wide-form by using pivot
quizwide = quizlong.pivot(index='ID',columns='Quiz',values='Score')
quizwide.reset_index().head()

# movie id | movie title | release date | video release date |
#              IMDb URL | unknown | Action | Adventure | Animation |
#              Children's | Comedy | Crime | Documentary | Drama | Fantasy |
#              Film-Noir | Horror | Musical | Mystery | Romance | Sci-Fi |
#              Thriller | War | Western |

#mv = pd.read_csv('http://files.grouplens.org/datasets/movielens/ml-100k/u.item',sep='|',header=None,encoding='latin-1')
#mv.tail()

#mv.columns= ['movie id' , 'movie title' , 'release date', 'video release date', 'IMDB URL' , 'Unknown', 'Action',
#              'Adventure' , 'Animation', "Children" , 'Comedy' , 'Crime' , 'Documentary' , 'Drama' , 'Fantasy' ,
#              'Film-Noir' , 'Horror' , 'Musical' , 'Mystery' , 'Romance' , 'Sci-Fi' , 'Thriller' , 'War' , 'Western']
#mv.head()

#mvlong = pd.melt(mv, id_vars=['movie id','movie title','release date','video release date','IMDB URL'],
#                   value_vars=['Unknown', 'Action', 'Adventure' , 'Animation', "Children" , 'Comedy' , 'Crime' ,
#                               'Documentary' , 'Drama' , 'Fantasy' , 'Film-Noir' , 'Horror' , 'Musical' , 'Mystery' ,
#                               'Romance' , 'Sci-Fi' , 'Thriller' , 'War' , 'Western'],
#                   var_name='Genre', value_name='Value')
#mvlong.head(10)

#mvlong.shape

#mv[mv['movie title'] == 'Star Wars (1977)']

#mvlong[mvlong['movie title'] == 'Star Wars (1977)']

# Which is the most popular movie genre?
# mvlong.groupby('Genre')['Value'].sum()


I am working on the file "Day 1 - Unit 2.3 Data Manipulations.ipynb".
The first task is:
Analysing categorical variables.
Can you write Python code for it?

# How many male and female users?
user.groupby('gender')['age'].count()

# What are the percentages of male and female users?
100*user.groupby('gender')['age'].count()/len(user)

# What are the percentages of male users for each occupation?
# First, create a new column to convert string to numbers/boolean
user['gen'] = user['gender'].map({'M': 1, 'F': 0})
user.head()

user.groupby(['occupation'])['gen'].mean()*100


I am working on the file "Day 1 - Unit 2.3 Data Manipulations.ipynb".
The first task is:
Challenge! Out of the top-100 most rated movies, which movie has the highest average rating?
Challenge!! Can you get the answer in one line of code?
Can you write Python code for it?


I am working on the file "Day 1 - Unit 2.3 Data Manipulations.ipynb".
The first task is:
Find the average and the number of ratings for all movies.
Can you write Python code for it?

df1 = df.groupby('movie_title')['rating'].agg([np.mean,np.size])
df1.head()


I am working on the file "Day 1 - Unit 2.3 Data Manipulations.ipynb".
The first task is:
Groupby method allows us to group rows of data together and call aggregate functions.
Can you write Python code for it?

# How many users for each occupation? Use of groupby method:
user.groupby('occupation').count()

# What is the average age for all users?
user['age'].mean()

# What is the average age for each occupation? Sort by descending order.
user.groupby('occupation')['age'].mean().sort_values(ascending=False)

# Get all summarized statistics about the age for each occupation.
user.groupby('occupation')['age'].describe()

# Apply customised function
# Example: Find the range of age for each occupation
user.groupby('occupation')['age'].apply(lambda x: x.max() - x.min())

# Calculate the mean age for each combination of occupation and gender.
user.groupby(['occupation','gender'])['age'].mean()
# Output is a multi-index series

# Selecting data in a multi-index series
user1 = user.groupby(['occupation','gender'])['age'].mean()
user1['student']['M']

# Unstacking the multi-index series
user.groupby(['occupation','gender'])['age'].mean().unstack()

# Unstacking multi-index series will give a data frame
user2 = user.groupby(['occupation','gender'])['age'].mean().unstack()
user2.loc['student', 'M']


I am working on the file "Day 1 - Unit 2.3 Data Manipulations.ipynb".
The first task is:
In this Unit 2.3, we will be using the MovieLens datasets: http://files.grouplens.org/datasets/movielens/ml-100k/README
Citation: F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets:
History and Context. ACM Transactions on Interactive Intelligent
Systems (TiiS) 5, 4, Article 19 (December 2015), 19 pages.
DOI=http://dx.doi.org/10.1145/2827872
2.3.1 Groupby: split-apply-combine
Import data set from url: http://files.grouplens.org/datasets/movielens/ml-100k/u.user
Description: Demographic information about MovieLens users. This is a tab separated list of user_id | age | gender | occupation | zip_code
Can you write Python code for it?

user = pd.read_csv('http://files.grouplens.org/datasets/movielens/ml-100k/u.user',sep='|',header=None)
user.columns = ['user_id','age','gender','occupation','zip_code']
user.tail()

# Any missing values?
user.isnull().sum()

# How many unique occupations are there in the data set?
user['occupation'].unique()

# The number of unique values.
user['occupation'].nunique()

# How many users for each occupation?
user['occupation'].value_counts()


I am working on the file "Day 1 - Unit 2.3 Data Manipulations.ipynb".
The first task is:
The apply method applies a callable to be applied to each column/entry of a dataframe/series.
Suppose that the zip_code can be classified as follows:
First digit of zip_code $\Rightarrow$ Region
| Digit | Region|
|:------:|:----:|
| 0, 1| North|
|2, 3| South|
|4, 5| East|
|6, 7| West|
|8, 9| Central|
Can you write Python code for it?

# Write a function to return a new column with 'Region'
def get_region(x):
    x = pd.to_numeric(x, errors='coerce')
    if x <= 1:
        return 'North'
    elif x <= 3:
        return 'South'
    elif x <= 5:
        return 'East'
    elif x <= 7:
        return 'West'
    elif x <= 9:
        return 'Central'
    else:
        return None

user['zip'] = user['zip_code'].str[0]
user['Region'] = user['zip'].apply(get_region)
user.head()

user.groupby(['Region','gender'])['age'].count().unstack()

def get_age_group(x):
    if x < 25:
        return 'Young Adults'
    elif x <= 60:
        return 'Adults'
    else:
        return 'Senior Citizens'

user['age_group'] = user['age'].apply(get_age_group)
user.head()

#user.groupby(['age_group','gender'])['age'].describe().columns


I am working on the file "Day 10 - Karatsuba multiplication.ipynb".
The first task is:
Convertion utility functions
Can you write Python code for it?

def int_to_big(x):
    if x == 0:
        return [0]

    z = []

    while x > 0:
        t = x % 10
        z.append(t)
        x //= 10

    trim(z)
    return z

def big_to_int(x):
    z, p = 0, 1

    for d in x:
        z += p * d
        p *= 10

    return z


I am working on the file "Day 10 - Karatsuba multiplication.ipynb".
The first task is:
Generate big integers
Can you write Python code for it?

def gen_long(n):
    x = ''.join(map(str, np.random.randint(0, 10, n)))
    return int(x)


I am working on the file "Day 10 - Karatsuba multiplication.ipynb".
The first task is:
Karatsuba multiplication using Baruchel's implementation
Karatsuba's algorithm is already implemented in Python. Check this package.
Can you write Python code for it?

from karatsuba import *

def power_of_two(x):
    p = 1
    while p < x:
        p <<= 1

    return p

def reverse(num):
    return int(str(num)[::-1])

def kat_multiply(x, y):
    if x == 0 or y == 0:
        return 0

    xs = list(map(int, str(x)))
    ys = list(map(int, str(y)))

    n = power_of_two(max(len(xs), len(ys)))
    plan = make_plan(range(n), range(n))

    xs = [0] * (n - len(xs)) + xs
    ys = [0] * (n - len(ys)) + ys

    zs = plan(xs, ys)
    zs.pop(-1)

    zs = list(reversed(zs))

    while zs[-1] == 0:
        zs.pop(-1)

    ans = 0
    for p, d in enumerate(zs):
        ans += d * 10 ** p

    return ans

tests = 30
for _ in range(tests):
    n = np.random.randint(1, 15)
    x, y = gen_long(n), gen_long(n)

    z = kat_multiply(x, y)

    assert z == x * y
    print("{} x {} = {}".format(x, y, z))

%%time

a, b = gen_long(100), gen_long(100)

z = kat_multiply(a, b)
assert z == a * b


I am working on the file "Day 10 - Karatsuba multiplication.ipynb".
The first task is:
Run(s)
Can you write Python code for it?


I am working on the file "Day 11 - Closest pair of points.ipynb".
The first task is:
Naive implementation of closest_pair
Can you write Python code for it?

def naive_closest_pair(points):
    best, p, q = np.inf, None, None
    n = len(points)

    for i in range(n):
        for j in range(i + 1, n):
            d = euclid_distance(points[i], points[j])

            if d < best:
                best, p, q = d, points[i], points[j]

    return best, p, q


I am working on the file "Day 11 - Closest pair of points.ipynb".
The first task is:
Run(s)
Can you write Python code for it?

points = [(26, 77), (12, 37), (14, 18), (19, 96), (71, 95), (91, 9), (98, 43), (66, 77), (2, 75), (94, 91)]

xs, ys = zip(*points)

d, p, q = closest_pair(points)
assert d == naive_closest_pair(points)[0]

print("The closest pair of points is ({0}, {1}) at distance {2}".format(p, q, d))

draw_points(points, p, q)

N = 10
x = np.random.rand(N) * 100
y = np.random.rand(N) * 100

points = list(zip(x, y))

d, p, q = closest_pair(points)
assert d == naive_closest_pair(points)[0]

print("The closest pair of points is ({0}, {1}) at distance {2}".format(p, q, d))

draw_points(points, p, q)

N = 20
x = np.random.randint(100, size=N)
y = np.random.randint(100, size=N)

points = list(zip(x, y))

d, p, q = closest_pair(points)
assert d == naive_closest_pair(points)[0]

print("The closest pair of points is ({0}, {1}) at distance {2}".format(p, q, d))

draw_points(points, p, q)

N = 20
x = np.random.rand(N)
y = np.random.rand(N)

points = list(zip(x, y))

d, p, q = closest_pair(points)
assert d == naive_closest_pair(points)[0]

print("The closest pair of points is ({0}, {1}) at distance {2}".format(p, q, d))

draw_points(points, p, q)


I am working on the file "Day 14 - Forward Euler.ipynb".
The first task is:
Example(s)
Can you write Python code for it?

print(forward_euler(lambda x, y: y + pow(math.e, x), 0, 1, 1, 4))

print(forward_euler(lambda x, y: y + pow(math.e, x), 0, 1, 1, 40)[-1])

e_approx = forward_euler(lambda x, y: pow(math.e, x), 0, 1, 1, 1000)

for it, e in enumerate(e_approx):
    print("Iteration:", it, math.e, e, abs(e - math.e))


I am working on the file "Day 14 - Forward Euler.ipynb".
The first task is:
The forward Euler method
$\frac{dy}{dx}=f(x,y) \quad y=y_0 \  when \ x=x_0 \quad x_0<x<X$
The forward Euler method calculates the values $y_1, y_2, y_3,... ,y_{N−1}, y_{N}$ using the formula
$y_i - y_{i−1} = \frac{f(x_{i-1},\ y_{i-1})}{h} \quad i = 1, 2, 3, . . . , N$
This may be written as the explicit formula $y_i = y_{i-1} + hf(x_{i-1},\ y_{i-1}) \quad i = 1, 2, 3,... , N$
Methods where an explicit expression for $y_i$ may be written down are known as explicit methods.
Can you write Python code for it?

def forward_euler(f, x0, y0, X, N):
    h = (X - x0) / N
    ys = [y0]

    for i in range(1, N + 1):
        ys.append(ys[-1] + h * f((i - 1) * h, ys[-1]))

    return ys


I am working on the file "Day 2 - Unit 3.1.ipynb".
The first task is:
1. Visualisation with  matplotlib

The library matplotlib is an extremely versatile graphics library written for Python in order to replicate and produce MATLAB type visualisations for engineering and scientific research. It is a low-level library and uses Python as the grammar for its graphics creation.
In this section of the course, we will learn about the Figure and Axes objects. These represent the backbone of matplotlib and all other objects are created on these two objects. We will learn how to generate the simplest chart - the "bread and butter" of scientific visualisation - line plots and scatter plots.
Then we will cover distributional plots in one and two dimensions. Density based plots are especially important when the number of data points are huge and overplotting becomes a problem.
Next we will learn how to create subplots, annotations and customize your plot.
Throughout this course, we will also learn and  put  some basic principles of data visualizations into practice.
1.1 Learning objectives
In this section we will:

Importing the matplotlib library and the %matplotlib inline magic command.
How to set up Figure and Axes objects.
Using the .plot method and its various keyword arguments.
Adding plot titles at the axis level.

1.2 Win, lose or DRAW!
By default, Jupyter Notebooks do not display matplotlib objects immediately after they are created. In order to activate the auto display function, we have to instruct Jupyter to do so by using the Ipython magic commands. In the cell below, run the command %matplotlib inline.
Can you write Python code for it?


I am working on the file "Day 2 - Unit 3.1.ipynb".
The first task is:
2.1 Using plt.plot to plot lines
Before we get to plotting the percentage difference in stock prices. Let's get a feel of how to plot lines in matplotlib. Like a scatterplot, to plot lines we:
  * Pass an array of x-coordinates and y-coordinates to the first and second arguments of plt.plot.
  * If we do not pass any other arguments, plt.plot thinks that you want to plot a line graph.
Can you write Python code for it?

# Use this space to do the following: Plot a simple line graph using plt.plot. Use the data below.

xcoord = [1,2,3,4]
ycoord = [5, -3, 9, -1]

# Answer:


I am working on the file "Day 2 - Unit 3.1.ipynb".
The first task is:
Check your current working directory. You should see the jpeg file there now.
Congratulations! You have completed your first visualisation project!
2. Basic plotting with .plot method

The basic workflow of creating a visualisation consists on instancing a figure and axes class, getting data and plotting it and finally saving your output to a file. In this section, we will learn about matplotlib's versatile .plot method to create basic scatter and line plots.
The function of the .plot method is to create scatter and line plots as have been mentioned. You have seen how to create basic scatter plots, so lets work through a simple visualisation project using line plots. Along the way, we will learn about the various customization and keyword arguments to the .plot method.
For our first task, we want to compare stock prime movements of Apple Corp. (AAPL), 3M Corp. (MMM) and Google (GOOGL) from the 9th of November 2016. Stock price data can be obtained from Google and pandas has an extension API to get this data from Google. We download the data, clean it and calculate the percentage difference for one time interval.
Can you write Python code for it?

import pandas as pd
import pandas_datareader.data as web # requires pandas-datareader package
from datetime import datetime

start = datetime(2016,11,9)
# end = datetime(2017, 7, 24)
tickers = ["AAPL", "MMM", "GOOGL"]
stock_panel = web.DataReader(tickers, "google", start, )

closing_pct_change = (stock_panel.loc["Close", :, :]
                                 .pct_change())

# closing_pct_change.to_csv("closing_pct_change.csv")
# backup plan
# closing_pct_change = pd.read_csv("closing_pct_change.csv")


I am working on the file "Day 2 - Unit 3.1.ipynb".
The first task is:
Here above, you will see our very first plot using the plt.plot function: A simple scatter plot consisting of coordinates $$(1,4), (2,7), (3, -1)$$ on the $x$-$y$ plane. Notice the arguments passed to plt.plot.

A list of the $x$-coordinates, [1,2,3]. In general purpose usage, this is a Series or an array object containing the predictor variables of our data.

A list of the $y$-coordinates, [4,7,-1]. This will usually be the response variable.

The final string bo is a convenience syntax for instructing plt.plot to create a scatterplot using round (o) markers coloured blue (b).

Note that if not marker string is passed to .plot, matplotlib will output a line plot by default.
All this is very nice, but before we go into more details about plt.plot, we must set up the general  workflow for working with this library.
1.3 The canvas: figure and axes
The parent object of any matplotlib visualisation is the Figure class. Figures contains the Axes objects which in turn contains all the other components that make up a plot: lines, patches, dots, text, arrows, contours and the like. Even objects like legends, colorbars, axis labels, ticks and plot titles are all subclassed from either a figure object or an axes object. Note that all visualisations produced by matplotlib are always contained in a Figure and axes. If these are not explicitly instanced, that they are implicitly created when plt.plot is called.
The matplotlib API can be interacted with using MATLAB style interfacing which uses function calls to plt.plot and various other annotation functions.
However, in this course, we wish to emphasize the object oriented interface to the API. This makes it easier  to customize matplotlib and even  seaborn visualisations. Essentially plot customization is about tweaking the attributes of an object.
To create a figure object:
Can you write Python code for it?


I am working on the file "Day 2 - Unit 3.1.ipynb".
The first task is:
In matplotlib, an axes can contain many different such lines by repeatedly calling the .plot method  on the same axis object. This has the effect of constructing multiple lines on the same plot and is useful for comparing various time series objects.
The code for accomplishing is below:
fig, ax = plt.subplots()

ax.plot(time_series_1, label=name_1)
ax.plot(time_series_2, label=name_2)
...
ax.plot(time_series_n, label=name_n)

We must first create a figure and axis object. And this can be accomplished together using plt.subplots() (and list unpacking).
However, look at the code above, is there anyway in which you can improve this?
2.2 Plotting multiple lines using the for loop
Yes, you probably guessed that we can use a for loop to repeatedly draw different time series objects on the same axes. By passing a unique name for each time series to the label keyword argument we are able to call legend to display the labels for each line.
In the cell below, we apply all that we have said above to plot the percentage change in closing stock price for Apple, 3M and Google on the same plot. We then place a legend on the plot and use .set_title method to write down a title for our plot.
The companies ticker codes are stored in a variable name tickers.  We will use the ticker codes  to access each column in the data frame by name.
The code to do all this is written below. Simply execute the cell below to see the results.
Can you write Python code for it?

fig2, ax2 = plt.subplots(figsize=(12,6)) # figure size is given in (wide, height) units

for ticker in tickers:
    ax2.plot(closing_pct_change[ticker], label=ticker)

plt.legend(loc="best")
ax2.set_title("Stock prices percentage change")


I am working on the file "Day 2 - Unit 3.1.ipynb".
The first task is:
Now we need to import the matplotlib plotting library. We use the import and as keyword to do this. The keyword as allows us to replace the lengthy module name with an alias of our choosing. By convention, use plt.
Can you write Python code for it?

import matplotlib.pyplot as plt

plt.plot([1,2,3], [4,7, -1], 'bo')


I am working on the file "Day 2 - Unit 3.1.ipynb".
The first task is:
The reason for initializing a figure instance calling plt.plot in the same cell is so that axes for this plot (call it ax1) will be contained in fig1.  By doing this we can save the figure to a file by running the following command
fig1.savefig("my_first_plot.jpeg", dpi=300)

in possibly another cell. Since this figure contains that ax1 object, the saved jpeg file will contain the plot above.  If the plot and figure instance were executed in different cells  your jpeg file will be empty when fig1.savefig is called!
You may want to rerender your plot to adjust some parameters. By putting the .savefig method in another cell, you create a jpeg file output only once after you are satisfied with what you see (in Jupyter Notebook).
Can you write Python code for it?


I am working on the file "Day 2 - Unit 3.1.ipynb".
The first task is:
When run, we see that an instance of a Figure class has been created. However, we do not see anything more that that printed output. In fact, if we were using another Python IDE like Spyder or accessing Python from the command prompt (like IPython), we will see a blank pop window created. Since we are using Jupyter Notebooks, our particular graphical renderer (inline) will not display this.
From this point on, we can create the plots with desire using plt.plot and assign the output to a variable name. Copy the line in the cell above and paste it below.
Can you write Python code for it?

                         # This line will be pasted in by course attendee.

plt.plot([1,2,3], [4,7,-1], '*r')


I am working on the file "Day 2 - Unit 3.2.ipynb".
The first task is:
3.1 Understanding the plotting code
So how was this produced?

We initialized a figure object using plt.figure.
Next we created an axis within the figure. We used the command fig.add_subplot. We passed an argument 111 to the function which is a short hand for creating only one axes.
With an ax object created, we now set a title using the set_title function and also label the x -axis with its label.
The histogram proper is plotting by calling the hist method on the axes instance. We merely need to pass the (1-dimensional) array of data to the function.

Notice the output that is produced. It's not very nice and probably needs some prettying up. But as a quick exploratory plot, it does its job. Notice the extra textual output. These can be suppressed with the ; written at the end of the last statement in the code block.
3.2 Customizing our histogram
The chart above can be customized to our liking by passing keyword arguments to the function.

Number of bins. The number of bins may be adjusted with the bins= keyword argument. However, do note that more bins does not translate into a better chart. With more bins, one tends to pick up much more variation between data (noise) that what is necessary. Therefore, try to choose a value which will give you the best sense of how the data is dristributed between the extremes of no details (small bins value) to a noisy chart (high bins value).

Normalization. Setting normed=True (it is False by default) means that the total area of the histogram is set to 1. This setting is useful to compare distributions of variables on different orders of magnitude.

Is a log scale needed?. log=True  may be used if you want the count (or relative counts) to be plotted on a log scale. This may be useful if the counts in different bins differ by huge orders of magnitude. This is especially true for data modelled by to power law distributions.

Cumulative sums. Sometimes, you want to plot the ogive instead. Enable this by setting cumulative=True.

Colors. Selecting the best...
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)

ax.set_title("Histogram of dew point readings")
ax.set_xlabel("Dew point (C$^\circ$)")
# Try typing in ax.hist(PRSA.DEWP, ...) with various customization options.


I am working on the file "Day 2 - Unit 3.2.ipynb".
The first task is:
As expected, bike rentals are low in bad weather. However, notice the variation within a week for the bad weather category. It is quite clear that people do not go biking on a bad weather Sunday since they have a choice not to go out!
Let's rerun the cell above by replacing the ax.hist function with the following code snippet. This helps us create a stacked bar chart.
ax2.set_ylim(*(0,5e5))
ax2.hist(day_data,
       label=weathers,
       weights=weights_data,
       bins=np.linspace(-0.5, 6.5, 8),
       ec="k", lw=1.,
       histtype="barstacked",
       rwidth=0.8
    )

A note on this new code: The histtype="barstacked" stacks the data on top each other. The new parameter rwidth= sets the ratio between the bar width and the bin width, which is the way you put spaces between bars. We disable log scale so that the natural totals are more clearly seen. Notice the extreme difference between rentals in different weather conditions.
There is hardly any differences between work days and weekends although we can detect a slight increase through the week. People do love the outdoors!
3.4 Summary of plotting a histogram
To summarize the teaching points above:

Pass either a single array of data to make into a histogram or a list of arrays if you want multiple data on one chart. You do not need to summarize the data. hist will do it for you.
Visual properties can be customize with keyword arguments like color, lw, ec, alpha, etc...
You can control whether to normalize the plot to have unit area by setting normed.
Choose an appropriate chart by setting histtype= to either bar or barstacked.
Pass frequency counts to the weights= parameter if you have a summarized bin count already.

4. hist2d and the hexbin plot
We use hist2d to visualize the joint distribution of bivariate data. When we expect correlation between two variables, a two dimensional histogram helps us reveal the structure of that relationship and avoids overplotting.
4.1 Plotting and customizing a 2D histogram
Let's return to the PRSA data...
Can you write Python code for it?

PRSA = PRSA.dropna() #drop missing data
fig3, ax3 = plt.subplots(figsize=(8,6))

ax3.grid(b="off")
ax3.set_facecolor("white")
ax3.set_title("PM2.5 readings distribution by temperature")
ax3.set_ylabel("$\mu g$/$m^3$")
ax3.set_xlabel("Daily temperature (C$^\circ$)")
ax3.hist2d(PRSA.TEMP, PRSA["pm2.5"],
           bins=55,
           cmin=5,
           range=[[-15,40],[0,600]],
           cmap="Blues");


I am working on the file "Day 2 - Unit 3.2.ipynb".
The first task is:
As one can see, there's not much one can say about the structure of the data because every point below 400 $\mu g/m^3$ is totally filled up with blue color.
It is here, that a density plot helps mitigate this problem. The central idea is that individual data points are not so important in as much as they contribute to revealing the underlying distribution of the data. In other words, for large amounts of data, we want to visualize the distribution instead of visualizing how individual datapoints are placed.
For this lesson, we will look at this data set and others to investigate the use of other plotting functions in matplotlib.
2.1 Learning objectives

To use histograms to visualize distribution of univariate data.
To  customize histograms.
To use 2D histogram plots and hexbin plots to plot 2D distributions.
To plot colorbars to annotate such plots

3. Histograms
Histograms are created using the hist command. We illustrate this using the PRSA data set. Let's say that we are interested in plotting the distribution of  DEWP.
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)

ax.set_title("Histogram of dew point readings")
ax.set_xlabel("Dew point (C$^\circ$)")

# Enter plotting code below here: ax.hist(PRSA.DEWP)


I am working on the file "Day 2 - Unit 3.2.ipynb".
The first task is:
However, hexbin plots differ from the square 2d histograms in more ways than the type of tiling used. We may use hexbin plots to investigate how a dependant variable depends on 2 independant variables. Just as we passed bin frequencies to the weights parameter in hist, we pass the third dependant variable to the C parameter in hexbin. That means, we can visualize a two dimensional surface embedded in a 3D space as an altitude map.
4.3.1 The hexbin C parameter
Let's investigate how PM2.5 pollutants vary with temperature and atmospheric pressure in the PRSA dataset. To do that type in (or copy paste) the following code
fig5, ax5 = plt.subplots(figsize=(8,6))
ax5.grid(b="off")
ax5.set_facecolor("white")
ax5.set_title("PM2.5 pollutants as a function of temperature\nand atmospheric pressure")
ax5.set_xlabel("Temperature (C$^\circ$)")
ax5.set_ylabel("Pressure (hPa)")
img = ax5.hexbin(PRSA.TEMP, PRSA.PRES, C=PRSA["pm2.5"],
    gridsize=(30, 20),
    cmap="rainbow")
cbar = plt.colorbar(img, ax=ax5)
cbar.set_label("$\mu$g/m$^3$")
Can you write Python code for it?


I am working on the file "Day 2 - Unit 3.2.ipynb".
The first task is:
If you observe, img is a tuple of length 4. The last entry of the tuple is the image data. Next, after the last line of ax3.hist command, add in the function
cbar = plt.colorbar(img[3], ax=ax3)

This means that we are now going to plot a color bar in ax3 (that's what ax=ax3 means) using the image data from our 2d histogram (which is why we must pass img[3] as an argument to plt.colorbar. We save the created colorbar instance in a variable named cbar so that we may further customize it.
To add a title to the colorbar, add in the following line:
cbar.set_label("Frequency")

This is what you should see if everything is done correctly.
4.3 hexbin plots
2D histograms create a square grid and visualize frequency counts using colors. Instead of using a square grid, we may also a hexagonal grid. As hexagons have more sides, this smooths out the resulting image. Let's see this effect with the same PRSA data set as in the hist2d plot.
Can you write Python code for it?

fig4, ax4 = plt.subplots(figsize=(9,6))

ax4.grid(b="off")
ax4.set_facecolor("white")
ax4.set_title("PM2.5 distribution by temperature")
ax4.set_ylabel("$\mu$g/$m^3$")
ax4.set_xlabel("Temperature (C$^\circ$)")
img = ax4.hexbin(PRSA.TEMP, PRSA["pm2.5"],
           gridsize=55,
           mincnt=5,
          # bins="log",
           cmap="Blues",
        )
cbar = plt.colorbar(img, ax=ax4)
#cbar.set_label("$\log($Frequency)")
cbar.set_label("Frequency")


I am working on the file "Day 2 - Unit 3.2.ipynb".
The first task is:
Source : https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.5+Data
Can you write Python code for it?

plt.plot( PRSA.TEMP, PRSA["pm2.5"], 'o', color="steelblue", alpha=0.5)
plt.ylabel("$\mu g/m^3$")
plt.title("PM 2.5 readings as a function of temperature (Celsius)")


I am working on the file "Day 2 - Unit 3.2.ipynb".
The first task is:
Source: https://archive.ics.uci.edu/ml/datasets/bike+sharing+dataset
As you can see in the dataset above, we need to plot weekday on the x-axis and have cnt as the y-axis. How can we plot this using hist? The problem is that if we pass the weekday array to hist, we end up counting the frequency of each day in the dataset!
To solve this, we pass the cnt variable as a seperate parameter to hist through the keyword argument weights.
Can you write Python code for it?

# Please run this cell before proceeding

group = bike_sharing.groupby("weathersit")
weathers = [w_r for w_r, _ in group]
day_data = [group.get_group(weather).weekday for weather in weathers]
weights_data = [group.get_group(weather).cnt for weather in weathers]


I am working on the file "Day 2 - Unit 3.2.ipynb".
The first task is:
This chart is more informative that a simple scatterplot. For one, we now know that there are two modes in the distribution of temperature and pollutant. Furthermore, there is more variation in pollution levels in the colder seasons as compared to warmer days.
Let's try to understand how this plot was created.
fig3, ax3 = plt.subplots(figsize=(8,6))

We initialize a figure object of width=8 units and height=6 units and an axis object to contain our histogram.
ax3.grid(b="off")
ax3.set_facecolor("white")
ax3.set_title("PM2.5 readings distribution by temperature")
ax3.set_ylabel("$\mu g$/$m^3$")
ax3.set_xlabel("Daily temperature (C$^\circ$)")

These codes set the axis grid to invisible and the background color to white. The rest are plotting information: plot titles and the units used on each axes.
ax3.hist2d(PRSA.TEMP, PRSA["pm2.5"],
           bins=55,
           cmin=5,
           range=[[-15,40],[0,600]],
           cmap="Blues");

Finally, this is the command to plot the histogram. Since this is a 2d histogram, we pass two arrays of data, first for the x-axis and then for the y-axis.

The bins= parameter is set to 55. That means there are 55 bins in each dimension. Again, too high a number leads to overfitting and creates a very noisy chart. So choose a suitable number so as not to lose too much information.
cmin= controls which frequency counts are displayed. If a particular frequency count is below cmin, it is not plotted.
The range parameter sets the expected range of the data in each dimension. Data points which are outside the range are considered outliers and not tallied.
Finally cmap controls the color scheme used to indicate frequency counts. There are many color schemes to choose from and all can be seen here. "Blues" is a type of color scheme known as a sequential color scheme. Use this for measures like frequency counts where the contrast between min and max is important.

4.2 Annotating plots with colorbar
This plot is still not perfect. For example, it would be nice to have an way to...
Can you write Python code for it?


I am working on the file "Day 2 - Unit 3.2.ipynb".
The first task is:
What we need to do is to group the data set by weather rating and create a list of array data to be passed to hist. We can do this efficiently using the groupby method on data frames and list comprehension statements. Now we have three lists: One for weather rating, one for the day of the week and one more for the daily bike rental counts.
We pass this to the hist function and pass a sequence of floats [-0.5, 0.5,..., 6.5] so that each bar is nicely centered on the tick mark. We also plot this count on a log scale (set log=True) because we expect quite a large difference between bike rental counts in good weather as compared to bad. Without this, it is very difficult to see any variation for rentals during bad weather.
Can you write Python code for it?

fig2 = plt.figure()

ax2 = fig2.add_subplot(111)
ax2.set_title("Distribution of daily bike rental counts by weather conditions", y=1.05)
ax2.set_xlabel("Day")
ax2.set_ylabel("Bike rental counts")
ax2.hist(       ,               # day_data
         label=         ,       # weathers
         weights=          ,    # weights_data
         log=      ,            # We want the y-axis to be on a log scale
         bins=np.linspace(-0.5,6.5,8),
         histtype="bar", # this is the default setting.
         ec="k", lw=1.0,
    )
ax2.legend(loc="lower right", title="Weather\nrating", fontsize="x-small");


I am working on the file "Day 2 - Unit 4.1 Visualising Numerical Data.ipynb".
The first task is:
4.1.1 Distplot
In matplotlib, we use plt.hist to plot histogram of a numerical variable. In Seaborn, we use the distplot, which by default, combines histogram with kde plot.
Kernel Density Estimatation (KDE) is a non-parametric way of estimating the density function of underlying data.
https://en.wikipedia.org/wiki/Kernel_density_estimation
Can you write Python code for it?

# Visualise the distribution of sepal length
sns.distplot(iris['sepal_length']);

# Show histogram only by removing the KDE layer
sns.distplot(iris['sepal_length'],kde=False);
# Notice the y-axis is count/frequency

# Rugplot draws a small vertical tick at each individual data point
sns.distplot(iris['sepal_length'],kde=False, rug=True);

# Customisations: change the bin size, rename axis labels, set title
ax = sns.distplot(iris['sepal_length'],kde=False,bins=10)
ax.set(xlabel='Measurement (cm)',ylabel='Frequency',title='Sepal Length of Iris');


I am working on the file "Day 2 - Unit 4.1 Visualising Numerical Data.ipynb".
The first task is:
4.1.2 Kde Plot
The default choice of kernel is Gaussian. There are other choices of kernels for univariate kde plot but only Gaussian kernel is available for bivariate kde plot.
Can you write Python code for it?

# Kde plot for one variable
sns.kdeplot(iris['sepal_length'],shade=True);

# Kde plot for two variables
sns.kdeplot(iris['sepal_length'],iris['sepal_width']);

# Customising kde plot
sns.kdeplot(iris['sepal_length'],iris['sepal_width'],shade=True,shade_lowest=False,n_levels=15);

# Forms 3 subsets of from iris dataframe
setosa = iris.loc[iris.species == "setosa"]
versicolor = iris.loc[iris.species == "versicolor"]
virginica = iris.loc[iris.species == "virginica"]
# Show 3 overlapping kde plots
sns.kdeplot(setosa.sepal_length, setosa.sepal_width, cmap="Reds")
sns.kdeplot(versicolor.sepal_length, versicolor.sepal_width, cmap="Greens")
sns.kdeplot(virginica.sepal_length, virginica.sepal_width, cmap="Blues");
#sns.regplot('sepal_length','sepal_width',data=iris,fit_reg=False,color='k');


I am working on the file "Day 2 - Unit 4.1 Visualising Numerical Data.ipynb".
The first task is:
4.1.3 Visualising Linear Relationship
Can you write Python code for it?

# Scatter plot with fitted regression line (by default)
sns.regplot('sepal_length','sepal_width',data=iris);

# Show scatter plot only, without the regression line
sns.regplot('sepal_length','sepal_width',data=iris,fit_reg=False);

# Insert vertical and horizontal lines
sns.regplot('sepal_length','sepal_width',data=iris,fit_reg=False)
plt.axvline(iris['sepal_length'].mean(),color='k',linestyle=':')
plt.axhline(iris['sepal_width'].mean(),color='k',linestyle=':');


I am working on the file "Day 2 - Unit 4.1 Visualising Numerical Data.ipynb".
The first task is:
Import data
Seaborn comes with built-in data sets. The following are most commonly used data sets:
 - iris
 - tips
 - titanic
 - exercise
Can you write Python code for it?

# import built-in data sets from seaborn
iris = sns.load_dataset('iris')
iris.sample(5)


I am working on the file "Day 2 - Unit 4.1 Visualising Numerical Data.ipynb".
The first task is:
Introducing Seaborn
Seaborn is a high-level data visualisation package based on Matplotlib.
https://seaborn.pydata.org/index.html
A major advantage of Seaborn is the ease to create informative statistical charts with appealing default aesthetics.
However, Seaborn should be viewed as a complement to matplotlib, not a replacement for it. A combination of both will allow you to create highly customisable charts with visually appealing aesthetics.
Content:
- 4.1.1 Distplot
- 4.1.2 Kdeplot
- 4.1.3 Visualising Linear Relationship
Import libraries
Can you write Python code for it?

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline


I am working on the file "Day 2 - Unit 4.1 Visualising Numerical Data.ipynb".
The first task is:
Use scipy.stats library to obtain the equation of the regression line: $\hat{y}=b_1 x + b_0$
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.linregress.html
Can you write Python code for it?

import scipy.stats as stats
# stats.linregress will return five outputs:
slope, intercept, r_value, p_value, std_err = stats.linregress(iris['sepal_length'],iris['sepal_width'])

# Print the regression line y=mx+c
reg_eqn = 'y = %.2f x + %.2f'% (slope,intercept)
print(reg_eqn)

# Display equation of regression line and R^2 value
sns.regplot('sepal_length', 'sepal_width', data=iris)
plt.text(6.5,4.2, reg_eqn, fontsize=10)
plt.text(6.5,4,'$R^2 = %.4f$' % (r_value**2), fontsize=10);

# Show residual plot
sns.residplot('sepal_length','sepal_width',data=iris);


I am working on the file "Day 2 - Unit 4.2 Visualising Categorical Data.ipynb".
The first task is:
4.2.2 Swarmplot
Swarmplot is similar to stripplot, but the points are adjusted automatically so that they don't overlap.
Can you write Python code for it?

sns.swarmplot(x='measurement',y='value',hue='species',data=irislong);

# Another example using Tips data set
sns.swarmplot(x="day", y="total_bill", data=tips);

sns.swarmplot(x="day", y="total_bill", hue='smoker', data=tips);


I am working on the file "Day 2 - Unit 4.2 Visualising Categorical Data.ipynb".
The first task is:
4.2.3 Boxplot
Boxplot can be applied to long-form or wide-form of data. But long-form is preferred to take advantage of hue parameter.
Can you write Python code for it?


I am working on the file "Day 2 - Unit 4.2 Visualising Categorical Data.ipynb".
The first task is:
4.2.5 Countplot
The traditional bar chart is called Countplot in Seaborn.
Can you write Python code for it?

# Count the frequency of each categorical variable
sns.countplot(x='day',data=tips);

# Display the count for each bar
ax = sns.countplot(x='day',data=tips);
for p in ax.patches:
    h = p.get_height()
    ax.text(p.get_x()+p.get_width()/2, h+2, '{:1.0f}'.format(h), ha="center")
ax.set_title('Number of customers per day');


I am working on the file "Day 2 - Unit 4.2 Visualising Categorical Data.ipynb".
The first task is:
4.2.7 Pointplot
Pointplot also shows the point estimate and CI like Barplot, but without the bar.
Can you write Python code for it?


I am working on the file "Day 2 - Unit 4.2 Visualising Categorical Data.ipynb".
The first task is:
Combining boxplot with swarmplot
Can you write Python code for it?


I am working on the file "Day 2 - Unit 4.2 Visualising Categorical Data.ipynb".
The first task is:
Combining violinplot with swarmplot.
Can you write Python code for it?


I am working on the file "Day 2 - Unit 4.2 Visualising Categorical Data.ipynb".
The first task is:
Import data
Can you write Python code for it?

tips = sns.load_dataset('tips')
tips.head()

# Iris data in wide-form
iris = sns.load_dataset('iris')
iris.head()

# Melt iris data to long-form
irislong = pd.melt(iris,id_vars='species', var_name='measurement')
irislong.head()


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
A Variable is a tensor which allows the addition of trainable parameters to the computational graph. Constants are intialized when created, as opposed to variables, which need to be initialized within the session (and the initialization procedure must be defined). Variables can be "manually" assigned a new value using tf.assign, and their state is kept within the session object. This is mostly used for model training, during which variables are changed within the optimization process.
Can you write Python code for it?

# create a variable, initialize it, and assign a new value within a session
sess = tf.Session()
a = tf.Variable(5)
print(a)
sess.run(tf.global_variables_initializer())
sess.run(a)
sess.run(tf.assign(a,6))
print(sess.run(a))

sess.close()

sess = tf.Session()
sess.run(tf.global_variables_initializer())
print(sess.run(a))


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
Exercise:

The model input $x$ is a placeholder for a data
The trainable model parameters $w$ and $b$ are defined as TensorFlow Variables
The model output $\hat{y}$ is a Tensor
The obesrved output $y$ is also a placeholder, where output data will be provided in order to train the model
Can you write Python code for it?

#define placeholders for the data
x = tf.placeholder(dtype=tf.float32,shape=[None,2])
y = tf.placeholder(dtype=tf.float32,shape=[None,1])

#define variables for the trainable parameters of the model
w = tf.Variable(tf.random_normal([2,1]),name="weights")
b = tf.Variable(tf.random_normal([1]), name="bias")

#create a tensor to calculate the model output
y_model = 1/(1+tf.exp(-(tf.matmul(x,w) + b)))

#define the loss function, create the optimizer and the training operation
loss = tf.reduce_mean(tf.square(y_model-y))
optimizer = tf.train.GradientDescentOptimizer(0.3)
train = optimizer.minimize(loss)

#train the model
sess.run(tf.global_variables_initializer())

for epoch in range(1000):
    sess.run(train,{x:x_train,y:y_train})
    print('w:',sess.run(w,{y:y_train,x:x_train}),', b:',sess.run(b,{y:y_train,x:x_train}),', loss:',sess.run(loss,{y:y_train,x:x_train}))


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
However, any Tensor is only evaluated within a Session, which is the environment in which all tensors and operations are executed.
Can you write Python code for it?

# create a TensorFlow session and evaluate the created constant
sess=tf.Session()
print(sess.run(t))


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
Inspect the trained model parameters and the model outputs. What is the minimum found by the optimizer?
Can you write Python code for it?


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
Linear regression in TensorFlow
Linear regression is one of the simplest and most commonly used regression models. The multivariate linear regression can be written as:
$$y = w^{T}x + b$$
where $y \in \mathbb{R}$ is the output, $w \in \mathbb{R}^{p}$ is a column vector containing $p$ weights for $p$ features in $x \in \mathbb{R}^{p}$, and $b \in \mathbb{R}$ is the bias. The parameters contained in $w$ and $b$ are also called coefficients and are trained by using a gradient descent algorithm.
Exercise:
Let us build a univariate linear regression model for a simple problem, using the previously introduced TensorFlow concepts:
- The model input $x$ is a placeholder for data
- The trainable model parameters $w$ and $b$ are defined as TensorFlow Variables
- The model output $\hat{y}$ is a Tensor
- The obesrved output $y$ is also a placeholder, where data will be provided for training purpose
Can you write Python code for it?

#define placeholders for data
x = tf.placeholder(dtype=tf.float32,shape=[None])
y = tf.placeholder(dtype=tf.float32,shape=[None])

#define model parameters as variables
w = tf.Variable(tf.random_normal(shape=()))
b = tf.Variable(tf.random_normal([]))

#create a tensor which calculates the model output
y_model = w*x + b


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
Logistic Regression
Logistic regression is a very common and simple linear model for classification purposes, based on linear regression and the logistic function:
$$y = \frac{1}{1+e^{-(w^{T}x + b)}}$$
Due to the nature of the logistic function, it produces output values in the range $[0,1]$, thus providing a probability for each class given in the output. Similar to linear regression, the variables defined within the logistic regression model are parameters trainable by various optimization algorithms.
Let us build a logistic regression for the well-known XOR problem.
Can you write Python code for it?

#generate XOR training data
import numpy as np
x_train = np.array([[0,0],[0,1],[1,0],[1,1]])
y_train = np.array([[0],[1],[1],[0]])

#import matplotlib for visualization
%matplotlib inline
import matplotlib.pyplot as plt

#logical indices of data where the outputs are 1 and 0
t = np.where(y_train==1)[0]
f = np.where(y_train==0)[0]

#scatter plot of the data
plt.scatter(x_train[t,0],x_train[t,1],c='b',marker='x',s=70)
plt.scatter(x_train[f,0],x_train[f,1],c='r',marker='o',s=70)


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
Multilayer Perceptron
A multilayer perceptron is a feedforward network that can be thought of a model composed of multiple nested functions, for instance:
$$y = f^{(3)}(f^{(2)}(f^{(1)}(x)))$$
This means that the output of each function is routed as the input of the next function, and this operational and data flow is strictly one-directional (thus "feedforward") and may contain multiple layers of nested functions (thus "deep"). TensorFlow is a very suitable tool for building and training such models. Here we will consider the XOR problem once again, and build a multilayer perceptron to classify the data correctly.
It was demonstrated previously that the XOR data are not linearly separable - this means that a non-linear layer (function) within the model is needed to tranform the problem to a linearly separable space. This is in fact the core of the multilayer perceptron as well as other deep learning models - nonlinear activation functions such as the logistic function, $tanh$, or ReLU. A comprehensive guide for TensorFlow supported functions can be found in: https://www.tensorflow.org/versions/r0.12/api_docs/python/nn/activation_functions_.
Let us build a multilayer perceptron model where the sigmoid activation function is used for the hiddern layer. Let:
- $f^{(1)}(x) = W^{(1)}x + b^{(1)}$
- $f^{(2)}(x) = {1}/({1+e^{-x}})$
- $f^{(3)}(x) = W^{(2)}x + b^{(2)}$
with $W^{(1)} \in \mathbb{R}^{2\times 2}$, $b^{(1)} \in \mathbb{R}^{2\times 1}$, $W^{(2)} \in \mathbb{R}^{2\times 1}$, and $b^{(2)} \in \mathbb{R}$.
Can you write Python code for it?

sess = tf.Session()

x_train = np.array([[0,0],[0,1],[1,0],[1,1]])
y_train = np.array([[0],[1],[1],[0]])

X = tf.placeholder(tf.float32,[None,2])
y = tf.placeholder(tf.float32,[None,1])

W1 = tf.Variable(tf.random_uniform([2,2]),name="weights1")
b1 = tf.Variable(tf.random_uniform([2]), name="bias1")

W2 = tf.Variable(tf.random_uniform([2,1]),name="weights2")
b2 = tf.Variable(tf.random_uniform([1]), name="bias2")

f1 = tf.matmul(X,W1)+b1
f2 = tf.nn.sigmoid(f1)
y_model = tf.matmul(f2,W2)+b2

loss = tf.reduce_mean(tf.square(y_model-y))

optimizer = tf.train.GradientDescentOptimizer(0.35)
#optimizer = tf.train.AdamOptimizer(0.1)
train = optimizer.minimize(loss)

sess.run(tf.global_variables_initializer())

for epoch in range(1000):
    sess.run(train, feed_dict={X: x_train, y: y_train})
    print("loss:", sess.run(loss,{X: x_train, y: y_train}))


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
Other very common and useful methods for creating tensors of constant value are tf.zeros() and tf.ones().
Can you write Python code for it?

# create a tensor of any shape populated with zeros and check within the session
t = tf.zeros([3,2,2])
print(sess.run(t))

# create a tensor of any shape populated with ones and check within the session
t = tf.ones([2,4])
print(sess.run(t))


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
Placeholders and Variables
Placeholders and Vairables are special kinds of tensors which are the essential building blocks of more complex data and computation streams. These are the most commonly used types of tensors in TensorFlow.
A Placeholder is a tensor which acts like a "promise" to provide a value at the evaluation of the computational graph. Placeholders are mostly used as input points in the computational graph where data will be provided. It will produce an error when evaluated, unless the value is fed to the session.
Can you write Python code for it?

# create a placeholder and feed it a value in a session
a = tf.placeholder(dtype=tf.float32)
b = tf.exp(a)
print(b)
print(sess.run(b,feed_dict={a:5}))

# create two placeholders and a tensor implementing matrix multiplication
x1 = tf.placeholder(dtype=tf.float32)
x2 = tf.placeholder(dtype=tf.float32)
y = tf.matmul(x1,x2)
print(sess.run(y,{x2:[[1,2],[3,4]],x1:[[1,2],[3,4]]}))


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
Simple algebraic operations such as +,-,/,and * can be used with tensors in this form, or by calling tf.add(), tf.subtract(), tf.divide(), or tf.multiply(). These are all element-wise, and defined for tensors of equal shapes and data-types. Tensors can be cast into a specific data type by calling tf.cast().
Can you write Python code for it?

# add a scalar to a tensor
a = tf.ones([3,2])
sess.run(a+3)

# subtract two tensors
a = tf.constant(4.,shape=[2,3])
print(a)
b = tf.random_normal(shape=[2,3])
print(b)
sess.run(a-b)

# divide two integer tensors
a = tf.constant(4,shape=[2,3])
b = tf.constant(7,shape=[2,3])
print(a/b)


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
Tensor
The basic concept behind TensorFlow is the tensor - an n-dimensional array of a base datatype. In TensorFlow it is represented by the tf.Tensor object which will produce a value when evaluated. A tf.Tensor object has a shape (which defines the structure of the elements) and a data type, shared by all the elements in the Tensor. The main types of tensors are:
- Constant
- Variable
- Placeholder
The tf.constant() method creates a constant tensor, populated with values of a data type, specified by arguments value, shape (optional), dtype (optional).
Can you write Python code for it?

# create a TensorFlow constant tensor
t = tf.constant(5)
print(t)

# create a TensorFlow constant of a specific data type and shape
t = tf.constant(7,shape=[2,3],dtype=tf.float32,name="const_tensor")
print(t)


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
Tensors containing random values from various distribution can be created using a number of methods, with the most commonly used being tf.random_uniform() and tf.random_normal().
Can you write Python code for it?

# create a random tensor containing values from a uniform distribution between 10 and 20
t = tf.random_uniform([3,4,2],minval=10,maxval=20)
print(t)
print(sess.run(t))


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow [SOLVED].ipynb".
The first task is:
The next layer $f^{(2)}(x)$ is the sigmoid function, which is a nonlinear transformation of the input, thus providing the possibility of transforming the problem to a new space where the outputs could be linearly separable.
Can you write Python code for it?

f2_out = sess.run(f2,{X: x_train, y: y_train})
print(f2_out)

plt.scatter(f2_out[t,0],f2_out[t,1],c='b',marker='x',s=70)
plt.scatter(f2_out[f,0],f2_out[f,1],c='r',marker='o',s=70)


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow.ipynb".
The first task is:
A Variable is a tensor which allows the addition of trainable parameters to the computational graph. Constants are intialized when created, as opposed to variables, which need to be initialized within the session (and the initialization procedure must be defined). Variables can be "manually" assigned a new value using tf.assign, and their state is kept within the session object. This is mostly used for model training, during which variables are changed within the optimization process.
Can you write Python code for it?

# create a variable, initialize it, and assign a new value within a session


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow.ipynb".
The first task is:
Exercise:

The model input $x$ is a placeholder for a data
The trainable model parameters $w$ and $b$ are defined as TensorFlow Variables
The model output $\hat{y}$ is a Tensor
The obesrved output $y$ is also a placeholder, where output data will be provided in order to train the model
Can you write Python code for it?

#define placeholders for the data

#define variables for the trainable parameters of the model

#create a tensor to calculate the model output

#define the loss function, create the optimizer and the training operation

#train the model


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow.ipynb".
The first task is:
Logistic Regression
Logistic regression is a very common and simple linear model for classification purposes, based on linear regression and the logistic function:
$$y = \frac{1}{1+e^{-(w^{T}x + b)}}$$
Due to the nature of the logistic function, it produces output values in the range $[0,1]$, thus providing a probability for each class given in the output. Similar to linear regression, the variables defined within the logistic regression model are parameters trainable by various optimization algorithms.
Let us build a logistic regression for the well-known XOR problem.
Can you write Python code for it?


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow.ipynb".
The first task is:
Other very common and useful methods for creating tensors of constant value are tf.zeros() and tf.ones().
Can you write Python code for it?

# create a tensor of any shape populated with zeros and check within the session

# create a tensor of any shape populated with ones and check within the session


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow.ipynb".
The first task is:
Placeholders and Variables
Placeholders and Vairables are special kinds of tensors which are the essential building blocks of more complex data and computation streams. These are the most commonly used types of tensors in TensorFlow.
A Placeholder is a tensor which acts like a "promise" to provide a value at the evaluation of the computational graph. Placeholders are mostly used as input points in the computational graph where data will be provided. It will produce an error when evaluated, unless the value is fed to the session.
Can you write Python code for it?

# create a placeholder and feed it a value in a session

# create two placeholders and a tensor implementing matrix multiplication


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow.ipynb".
The first task is:
Tensor
The basic concept behind TensorFlow is the tensor - an n-dimensional array of a base datatype. In TensorFlow it is represented by the tf.Tensor object which will produce a value when evaluated. A tf.Tensor object has a shape (which defines the structure of the elements) and a data type, shared by all the elements in the Tensor. The main types of tensors are:
- Constant
- Variable
- Placeholder
The tf.constant() method creates a constant tensor, populated with values of a data type, specified by arguments value, shape (optional), dtype (optional).
Can you write Python code for it?

# create a TensorFlow constant tensor

# create a TensorFlow constant of a specific data type and shape


I am working on the file "Day-1/First day - Introduction to Machine Learning with Tensorflow.ipynb".
The first task is:
To train a model built in TensorFlow, a loss function needs to be defined, most commonly as a reduction operation. An optimizer object needs to be defined, and the minimize() method called in order to update the variables defined within the model to minimize the selected loss function. When creating optimizer objects, choices about the learning rate have to be made - these, in combination with the number of training epochs, can greatly influence the model training process. With the approapriate learning rate, the optimization can quickly converge.
Can you write Python code for it?

#define the loss function as the mean of all squared errors (MSE)

#create a gradient descent optimizer

#create a train operation

#generate data to train the regression

#initialize variables, run 100 epochs of training algorithm


I am working on the file "Day-2/classification/mnist-solved.ipynb".
The first task is:
Every MNIST sample has two parts:
an image (vectorized, raster-scanned) of a handwritten digit and a corresponding label.
Can you write Python code for it?


I am working on the file "Day-2/classification/mnist-solved.ipynb".
The first task is:
Inputs
Here we specify placeholders for the TF computational graph. Basicall, this determines how to input images, $x$, and labels, $y$, into the learning process.
Can you write Python code for it?

def build_inputs():
    # `x` is a batch of input images (each reshaped into a vector: 28x28 -> 784)
    x = tf.placeholder(tf.float32, [None, 784])
    # `y` is a batch of labels
    y = tf.placeholder(tf.float32, [None, 10])
    # return for further processing
    return x, y


I am working on the file "Day-2/classification/mnist-solved.ipynb".
The first task is:
Learning the model parameters from data
Initially, $\mathbf{W}$ and $\mathbf{b}$ contain random values that will not produce correct classification results.
We have to tune these tensors by minimizing an appropirate loss function that will "measure" the quality of classification.
We will use the cross entropy criterion:
$$
    L(\mathbf{x}, c)=
    -\log p_c(\mathbf{x})
    ,
$$
where $p_c(\mathbf{x})$ is the probability assigned by the model that $\mathbf{x}$ belongs to class $c$,
$$
    p_c=
    \frac{e^{l_c}}{\sum_{j=1}^{10} e^{l_j}}
    ,
$$
and $(l_0, l_1, \ldots, l_9)=M(\mathbf{x})$ are the logits output by the model.
The derivatives can now be computed by TensorFlow and the model can be tuned with stochastic gradient descent ($k=0, 1, 2, \ldots$):
$$
    \mathbf{W}{k+1}=
    \mathbf{W}_k - \eta\frac{\partial L}{\partial\mathbf{W}_k}
$$
$$
    \mathbf{b}{k+1}=
    \mathbf{b}_k - \eta\frac{\partial L}{\partial\mathbf{b}_k}
$$
Can you write Python code for it?

def build_loss(logits, y):
    # labels `y` are one-hot encoded class indicators (`c` in the text above)
    loss = tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=logits)
    return loss


I am working on the file "Day-2/classification/mnist-solved.ipynb".
The first task is:
Our first classification model: softmax regression
We're going to train a model to look at images and predict what digits they are.
A function $M: \mathbb{R}^{28\times 28}\rightarrow \mathbb{R}^{10}$ outputs a classification score for each input digit.
In other words, $M(\text{image})=\text{a vector of per-class scores}$.
We want that a higher score for class $c$ translates to higher confidence that $c$ is the correct class.
For example, if $M$ outputs
$$
    (0.05, 0.03, 0.82, 0.02, 0.01, 0.02, 0.01, 0.02, 0.01, 0.1)
$$
for an input image, it classifies that image as a $2$.
Let us choose a very simple classification model first:
$$
    M(\mathbf{x})=
    \mathbf{x}\cdot\mathbf{W} + \mathbf{b}
    ,
$$
where $\mathbf{x}\in\mathbb{R}^{784}$ is a vectorized input image, and $\mathbf{W}\in\mathbb{R}^{784\times 10}$ and $\mathbf{b}\in\mathbb{R}^{10}$ are the model parameters. The elements of $M(\mathbf{x})$ are sometimes called logits.
Can you write Python code for it?

def build_affine(x):
    W = tf.Variable(tf.random_normal([784, 10], mean=0.0, stddev=0.01))
    b = tf.Variable(tf.zeros([10]))
    logits = tf.matmul(x, W) + b
    return logits


I am working on the file "Day-2/classification/mnist-solved.ipynb".
The first task is:
The MNIST dataset
Contains images of handwritten digits (0, 1, 2, ..., 9).
Download the dataset using TF's built-in method.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)


I am working on the file "Day-2/classification/mnist-solved.ipynb".
The first task is:
The loss $L$ is usually approximated on a batch of images.
The code above can handle this case as well.
We set the batch size to $100$ is our experiment.
Measuring the test set accuracy
We measure the quality of the model on a separate testing dataset by counting the number of images that it has correctly classified:
$$
    \text{accuracy}=
    \frac{\text{number of correctly classified samples}}{\text{total number of samples}}
$$
Can you write Python code for it?

def build_ncorrect(logits, y):
    # get correct predictions as a vector of bools (0s and  1s)
    correctpreds = tf.equal(tf.argmax(logits, 1), tf.argmax(y, 1))
    # sum the zeros and ones
    ncorrect = tf.reduce_sum(tf.cast(correctpreds, tf.float32))
    # return for future processing
    return ncorrect

def get_accuracy(ncorrect):
    acc = 0.0
    for i in range(0, 100):
        acc = acc + sess.run(ncorrect, feed_dict={x: mnist.test.images[i*100:(i+1)*100, :], y: mnist.test.labels[i*100:(i+1)*100, :]})
    acc = 100*acc/10000
    return acc


I am working on the file "Day-2/classification/mnist-solved.ipynb".
The first task is:
Trainig/testing loop
The code for traing and validating the model follows.
Can you write Python code for it?

def run_training_loop(step, ncorrect, batchsize, niters):
    #
    for k in range(niters):
        X, Y = mnist.train.next_batch(batchsize)
        sess.run(step, feed_dict={x: X, y: Y})
        if k % 500 == 0:
            acc = get_accuracy(ncorrect)
            print('* iter %d: test set accuracy=%.2f %%' % (k, acc))


I am working on the file "Day-2/classification/mnist.ipynb".
The first task is:
Our first classification model: softmax regression
We're going to train a model to look at images and predict what digits they are.
A function $M: \mathbb{R}^{28\times 28}\rightarrow \mathbb{R}^{10}$ outputs a classification score for each input digit.
In other words, $M(\text{image})=\text{a vector of per-class scores}$.
We want that a higher score for class $c$ translates to higher confidence that $c$ is the correct class.
For example, if $M$ outputs
$$
    (0.05, 0.03, 0.82, 0.02, 0.01, 0.02, 0.01, 0.02, 0.01, 0.1)
$$
for an input image, it classifies that image as a $2$.
Let us choose a very simple classification model first:
$$
    M(\mathbf{x})=
    \mathbf{x}\cdot\mathbf{W} + \mathbf{b}
    ,
$$
where $\mathbf{x}\in\mathbb{R}^{784}$ is a vectorized input image, and $\mathbf{W}\in\mathbb{R}^{784\times 10}$ and $\mathbf{b}\in\mathbb{R}^{10}$ are the model parameters. The elements of $M(\mathbf{x})$ are sometimes called logits.
Can you write Python code for it?


I am working on the file "Day-2/classification/mnist.ipynb".
The first task is:
Trainig/testing loop
The code for traing and validating the model follows.
Can you write Python code for it?

def run_training_loop(step, accuracy, batchsize, niters):
    #


I am working on the file "Day-2/classification/mnist.ipynb".
The first task is:
Training #1
Can you write Python code for it?

# prepare computation graphs
# (insert code below)

# final preparations for learning
sess = tf.Session()
tf.global_variables_initializer().run(session=sess)

# start the learning process: batch size=100, number of iterations=10000
# (insert code below)

# clear the current session (so we can start another one later)
sess.close()
tf.reset_default_graph()


I am working on the file "Day-2/segmentation/semantic_segmentation_clean.ipynb".
The first task is:
3. Training the model
Training
During training we are going to compute the forward pass first to get the value of the loss function.
After that we are doing the backward pass and computing all gradients the loss wrt parameters at each layer with backpropagation.
Can you write Python code for it?

# this functions trains the model
def train(sess, x, y, y_pred, loss, checkpoint_dir):
    num_epochs = 30
    batch_size = 10
    log_dir = 'local/logs'
    utils.clear_dir(log_dir)
    utils.clear_dir(checkpoint_dir)

    learning_rate = 1e-3
    decay_power = 1.0

    global_step = tf.Variable(0, trainable=False)
    decay_steps = num_epochs * train_data.num_batches
    # usually SGD learning rate is decreased over time which enables us
    # to better fine-tune the parameters when close to solution
    lr = tf.train.polynomial_decay(learning_rate, global_step, decay_steps,
                                   end_learning_rate=0, power=decay_power)

    ...

sess = tf.Session()
train(sess, x, y, y_pred, loss, 'local/checkpoint1')


I am working on the file "Day-2/segmentation/semantic_segmentation_clean.ipynb".
The first task is:
Dataset
The Dataset class implements an iterator which returns the next batch data in each iteration. Data is already normalized to have zero mean and unit variance. The iteration is terminated when we reach the end of the dataset (one epoch).
Can you write Python code for it?

batch_size = 10
num_classes = Dataset.num_classes
# create the Dataset for training and validation
train_data = Dataset('train', batch_size)
val_data = Dataset('val', batch_size, shuffle=False)

print('Train shape:', train_data.x.shape)
print('Validation shape:', val_data.x.shape)

#print('mean = ', train_data.x.mean((0,1,2)))
#print('std = ', train_data.x.std((0,1,2)))


I am working on the file "Day-2/segmentation/semantic_segmentation_clean.ipynb".
The first task is:
Inputs
First, we will create input placeholders for Tensorflow computational graph of the model. For a supervised learning model, we need to declare placeholders which will hold input images (x) and target labels (y) of the mini-batches as we feed them to the network.
Can you write Python code for it?

# store the input image dimensions
height = train_data.height
width = train_data.width
channels = train_data.channels

# create placeholders for inputs
def build_inputs():
    ...


I am working on the file "Day-2/segmentation/semantic_segmentation_clean.ipynb".
The first task is:
Loss
Now we are going to implement the build_loss function which will create nodes for loss computation and return the final tf.Tensor representing the scalar loss value.
Because segmentation is just classification on a pixel level we can again use the cross entropy loss function \(L\) between the target one-hot distribution \( \mathbf{y} \) and the predicted distribution from a softmax layer \( \mathbf{s} \). But compared to the image classification here we need to define the loss at each pixel. Below are the equations describing the loss for just one example (one pixel in our case).
$$
L = - \sum_{i=1}^{C} y_i log(s_j(\mathbf{x})) \
s_i(\mathbf{x}) = \frac{e^{x_i}}{\sum_{j=1}^{C} e^{x_j}} \
$$
Can you write Python code for it?

# this funcions takes logits and targets (y) and builds the loss subgraph
def build_loss(logits, y):
    ...


I am working on the file "Day-2/segmentation/semantic_segmentation_clean.ipynb".
The first task is:
Model
Now we can define the computational graph. Here we will heavily use tf.layers high level API which handles tf.Variable creation for us. The main difference here compared to the classification model is that the network is going to be fully convolutional without any fully connected layers. Brief sketch of the model we are going to define is given below.
conv3x3(32) -> 4 x (pool2x2 -> conv3x3(64) -> conv3x3(64)) -> conv1x1(7) -> resize_bilinear -> softmax() -> Loss
Can you write Python code for it?

# helper function which applies conv2d + ReLU with filter size k
def conv(x, num_maps, k=3):
    ...

# helper function for 2x2 max pooling with stride=2
def pool(x):
    ...

# this functions takes the input placeholder and the number of classes, builds the model and returns the logits
def build_model(x, num_classes):
    ...


I am working on the file "Day-2/segmentation/semantic_segmentation_clean.ipynb".
The first task is:
Putting it all together
Now we can use all the building blocks from above and construct the whole forward pass Tensorflow graph in just a couple of lines.
Can you write Python code for it?

# create inputs

# create model

# create loss

# we will need argmax predictions for IoU