I am working on the file "A Beginners Guide to Python/21. The joy of fast cars.ipynb".
The first task is:
In the above output, we can see that it took about 32secs to get all the primes less than 100,000.
We can also see that almost all of the time is taken by the "is_prime" function (tottime column). This is important, this means that improving the performance of "is_prime" is going to considerably increase performance, whereas improving the speed of the get_primes function will have almost no impact.
For example, list comprehensions are faster than for-loops and so I could speed the ‘get_primes’ function by doing that. For arguments sake lets suppose using a list comprehension can speed it up that function by a massive 90%!! That’s a huge improvement! But when we look at the total time we see that get_primes took 0.015 seconds on my machine. So a 90% improvement would speed us up by about 0.012 seconds.
Okay, so the function we need to improve is the ‘is_prime’ function. I think the first line of code to study is the for-loop:

for i in range(2, num):

So this is where the fun begins! To solve this puzzle we need to think logically and be a bit creative here. Improving the speed of this line of code is not simply “know more Python”. Rather, we need to think logically and apply a splash mathematics. Here, let me show you something:
Can you write Python code for it?

for i in range(1, 20):
    print(i, "--", 20/i)


I am working on the file "A Beginners Guide to Python/21. The joy of fast cars.ipynb".
The first task is:
Okay cool, now that we have working solution the next question is how to improve the speed. To improve speed, the first and most obvious starting point is to time the code.
Lets do that now...
Can you write Python code for it?

import sys
sys.path.append(".\misc") # Adding to sys.path allows us to find "profile_code.py"
from profile_code import profile

result = profile(get_primes, 100_000) # Get all primes, 0..100,000

print(result)


I am working on the file "A Beginners Guide to Python/21. The joy of fast cars.ipynb".
The first task is:
Okay, so it seems faster; 0.072 is less than 0.153. However, when you see very small difference in time the result can be unreliable. Because background process and other things happening on the computer may be affecting the results. The simplest way to check that we really do have a genuine difference it to increase our input size. I know! Lets get all the prime numbers less than 10 million!
Can you write Python code for it?

import sys
sys.path.append(".\misc") # Adding to sys.path allows us to find "profile_code.py"
from profile_code import profile

result = profile(get_primes4, 10_000_000) # Get all primes, 0..10,000,000

print(result)

import sys
sys.path.append(".\misc") # Adding to sys.path allows us to find "profile_code.py"
from profile_code import profile

result = profile(sieve_of_eratosthenes, 10_000_000) # Get all primes, 0..10,000,000

print(result)


I am working on the file "A Beginners Guide to Python/21. The joy of fast cars.ipynb".
The first task is:
So now we are down to 8secs.
Can we do better? Perhaps, but I’m out of ideas at this point. But hey, google is a treasure trove of information, I wonder if there is some other maths ‘trick’ out there we could use…
After googling, I found out that apparently we can use the square root of n! And here I’m going to reproduce a maths proof which I found here.
Proof
Imagine we have two numbers A, B such that A * B = N
Now there are three possible cases.

A > B
A = B
A < B

Notice that because A * B is the same as B * A cases 1 and 3 are equivelent. So that means we only need to check cases 1 and 2.
If A = B then A * B can be rewritten as A * A. If A * A = N then thats the definition of square root.
Benchmarking Sqrt(N)
Alright, how can we implement this. Well, it took a bit of testing, but eventually I came up with this line (after importing the math module, of course):

for num in range(3, math.ceil(math.sqrt(num))+1, 2):

Sqrt(n) in many cases is not a whole number in some cases and as discussed elsewhere range requires an integer. That’s where math.ceil comes in, it rounds n up to the next integer (eg. math.ceil(6.0003) ---> 7) I then add one to make sure small numbers like 3,4 are handled correctly.
How much faster do we think this function will be? Lets run it!
Can you write Python code for it?

# Attempt 4

from math import sqrt, ceil

def is_prime4(num):
    """Returns True if number is prime, False otherwise"""

    if num <= 1:
        return False
    if num == 2:
        return True
    if num % 2 == 0:
        return False

    # check for factors
    for i in range(3,ceil(sqrt(num))+1, 2): # upto sqrt of N (always rounded up)
                if (num % i) == 0:
                    return False
    return True

# Making the list:
def get_primes4(b):
    primes = []
    for num in range(0, b+1):
        if is_prime4(num):  ## call our new prime function...
            primes.append(num)
    return primes

import sys
sys.path.append(".\misc") # Adding to sys.path allows us to find "profile_code.py"
from profile_code import profile

result = profile(get_primes4, 100_000) # Get all primes, 0..100,000

print(result)


I am working on the file "A Beginners Guide to Python/21. The joy of fast cars.ipynb".
The first task is:
So this code is dividing the number 20 by 'i', where 'i' is 1-to-20. The salient point here is that numbers past 11 are not whole numbers. This makes a lot of sense when you think about it; the minimum number of ‘integer parts’ we can split X into (besides 1) is two. Thus, when we start looking at numbers greater than n/2 the solution will never be a whole number. And that stands for all numbers, not just 20.
Now, we can we use this information to make our prime search smarter. As things currently stand proving 1499 is prime requires about that many steps; our code is (at the moment) asking if numbers like 1001, 1002, 1003, ... are divisors of 1499 but as the above logic demonstrates since we want to find divisors of n these checks are actually unnecessary. So, If we stop iterating at the number 750 we can approximately half the time it takes to find a prime number and still have a correct solution.

for i in range(2, num//2):

As a quick note, we are using integer division here because the range function cannot handle floats. Now, before we run the benchmark though, we need to check for correctness; whenever you make a changes, even If it is a small one, you should test it on a few inputs. We want to check we haven’t broken anything with our change (more on ‘regression testing’ later). With this in mind, I ran the following code on my machine (where is_prime is the old function and is_prime2 is with the change):
x = [i for i in range(0,30000) if is_prime(i)]
y = [i for i in range(0,30000) if is_prime2(i)]
print(x == y) ---> False

We have a bug batman! What went wrong?  To find out, I ran the following bit of code:
x2 = set(x)
y2 = set(y)
x2.symmetric_difference(y2) ---> {4}

I converted the lists to sets because sets have this handy method for quickly telling the difference between two items. It turns out we have two lists, each with 3200+ numbers and the only difference is that one of these lists contains the number 4 and the other does not. So what’s the problem?
Well, our new function uses:
...
Can you write Python code for it?

n = 4
for i in range(2, n//2):
    print("(1)...", i)
    # Nothing is printed! WTF!!

# Okay, attempted fix:
for i in range(2, n//2+1):
    print("(2)..." , i)

# Attempt 2

def is_prime2(num):
    """Returns True if number is prime, False otherwise"""

    if num <= 1: return False

    # check for factors
    for i in range(2, (num//2) + 1):  ## tweaked
        if (num % i) == 0:
            return False

    return True

# Making the list:
def get_primes2(b):
    primes = []
    for num in range(0, b+1):
        if is_prime2(num):  ## call our new prime function...
            primes.append(num)
    return primes

import sys
sys.path.append(".\misc") # Adding to sys.path allows us to find "profile_code.py"
from profile_code import profile

result = profile(get_primes2, 100_000) # Get all primes, 0..100,000

print(result)


I am working on the file "A Beginners Guide to Python/21. The joy of fast cars.ipynb".
The first task is:
So this small change has roughly halved the amount of time it to get all primes upto 100,000.
Are we done? Well actually I can think of a few more tweaks...
Let's think about the nature of primes for one moment. The definition of a prime is that it is only divisible by itself and 1. And since an even number is, by definition, divisible by 2 we know that the only prime that is even is 2.
Let’s  think of a large odd number (not necessarily prime). Our code is going to ask if 2,4,8,10,12… are divisors. But from the definition of even numbers we know that if 12, 18, 22, etc are divisors of X then so must 2. Which therefore means if 2 is not a divisor then neither is 6,8,100,102, etc.
In short, checking for 2 is equivalent to checking for all even numbers. Can we apply this insight to our code? I think so:
Can you write Python code for it?

# Attempt 3

def is_prime3(num):
    """Returns True if number is prime, False otherwise"""

    if num <= 1:
        return False
    if num == 2:
        return True
    if num % 2 == 0:
        # notice that this check occurs AFTER we check is num == 2.
        return False

    # check for factors
    for i in range(3,num//2+1, 2): # range function starts at odd number with a step of 2.
                if (num % i) == 0:
                    return False
    return True

# Making the list:
def get_primes3(b):
    primes = []
    for num in range(0, b+1):
        if is_prime3(num):  ## call our new prime function...
            primes.append(num)
    return primes


I am working on the file "A Beginners Guide to Python/21. The joy of fast cars.ipynb".
The first task is:
So when we check for all primes less than 10 million the sieve_of_eratosthenes really stands out; 9 secs versus 66 sec for our previous champion.
Anyway, the main lesson I want you to learn here is that optimisation can be thought of as being two seperate ideas; the first idea is low-level tinkering, in other words, we look at all the small details and see if we can save a byte or two of memory here or there. But then there is ‘high-level’ optimisation, and that is where we try to come up with an entirely new (and hopefully better) strategy for solving the problem.
In this lecture we started searching for needles in haystacks (checking if each number was a prime). We improved that by checking fewer straws. And then we thought of new idea, which was a bit like searching for needles by setting the haystack on fire; with the idea being that the only thing left will be the needles (i.e. the primes).
Can you write Python code for it?

# Quick check to ensure all the functions return the correct answer..

a = get_primes(1000)
b = get_primes2(1000)
c = get_primes3(1000)
d = get_primes4(1000)
e = sieve_of_eratosthenes(1000)

a == b == c == d == e


I am working on the file "A Beginners Guide to Python/21. The joy of fast cars.ipynb".
The first task is:
When we started, it took 32secs to get all the primes from 0 to 100_000 (on my pc). With the square root trick it now takes 0.15secs, thats a massive difference.
You might wonder why square root made such a big difference. Why is to do with how functions scale with input size. For example, If we skip all the even numbers less than N, and we skip all the numbers greater than N / 2 then we have reduced the search space to approximately N / 4. This is a linear improvement; if the number is 1,000,000 then we still need to check 250,000 numbers. For 2,000,000 we check 500,000 numbers, and so on.
What about square root? Well the square root of a 1,000,000 is 1000 and the sqrt of 2,000,000 is about 1414. So as you can see, these numbers are growing at a much slower rate. And thats why its considerably faster. For a more theoretical explanation, please google "Big O Notation".
Are we done?
Rethinking the problem
So, our task is create lists of primes upto N and so far our strategy for improving performance is to reduce the number of divisors we need to check.
Lets think outside the box for a moment. Image that there is an empty swimming pool full of small plastic balls and equally sized lead balls. We want to sort them. How might we do it?
Well, we could jump into the ball and pick the balls up. If its heavy we put it to oneside. Maybe we could improve this process by getting a friend to help. Maybe we get fifty friends, and at some point we realise that the way to improve performance is to manage people better (i.e. teamwork), and so you go down a rabbit-hole of small incremental improvements.
But then a new idea comes along! Its a solution that doesn't need fifty people, in fact its a solution whose running time is independant of the number of people working on the task. Can you guess what the idea maybe?
Okay here it is; fill the pool with water, the lead balls will stay at the bottom but the plastic balls will float. Get a big net and voila! The balls are sorted.
How can we apply this analogy to the current...
Can you write Python code for it?

def sieve_of_eratosthenes(n):

    mark = [True] * (n)
    mark[0] = False
    mark[1] = False # 0 and 1 are not prime, so we set these values to false.

    i = 2 # start at index 2 since 2 is the smallest prime.
    while i < len(mark):
        p = i
        i += 1
        if not mark[p]:
            ## Ignore non-primes
            continue

        multiplier = 2
        while p * multiplier < n:
            ## Set all the multiples of P to false (since they cannot be prime).
            mark[p * multiplier] = False
            multiplier += 1

    return [i for i in range(len(mark)) if mark[i]] ## If all True values in the 'mark' array

import sys
sys.path.append(".\misc") # Adding to sys.path allows us to find "profile_code.py"
from profile_code import profile

result = profile(sieve_of_eratosthenes, 100_000) # Get all primes, 0..100,000

print(result)


I am working on the file "A Beginners Guide to Python/22. Common Error Messages.ipynb".
The first task is:
"==" is not "="
For experienced users, this is merely another type of typo. However, for beginners this error is sometimes more indicative of a more serious and fundamental misunderstanding.
Essentially the error here would be confusing assignment ("=") with asking if a equals b ("=="). If you do not understand this crucial distinction I'd strongly recommend revisiting the lectures on assignment and logic.
Here is a simple example:
Can you write Python code for it?

a == 8  # NameError; a is not defined!
print(a)

# The Fix:
a = 8
print(a)


I am working on the file "A Beginners Guide to Python/22. Common Error Messages.ipynb".
The first task is:
Common causes:

Another part of the code is misbehaving!
Misunderstanding properties of data-types and/or how thier methods work.
Oversights...

Problems elsewhere...
If you receive a type error, in may cases the direct cause usually isn't the problem, rather, the problem happened much earlier and you are just finding out about it now. What I mean is, everyone knows you cannot divide strings by strings and so its unlikely you wrote a piece of code to do just that. Rather, some other bit of code returned strings instead of integers and that mistake gets passed on to the next function.
For example...
Can you write Python code for it?

# Building a calculator !
print("Welcome to my amazing calculation machine 1.0!",
      "Give me two numbers and I add them together and tell you if the result is a perfect square", sep="\n")

a = input("please enter a number ")
b = input("and another number ")

def add(a, b):
    return a + b

def is_square(x):
    import math
    return math.sqrt(x).isinteger()

x = add(a, b)
print(is_square(x))


I am working on the file "A Beginners Guide to Python/22. Common Error Messages.ipynb".
The first task is:
Issues with Scoping
As I've mentioned in previous lectures, Python works in code "blocks" and such blocks have their own space to play with. Within that space variables can be defined and those variables are not affected (or even known) by other parts of the code. Common fixes include changing indentation levels, or saving variables to other names spaces. For example:
Can you write Python code for it?

def func():
    t = 35
print(t)

# Possible Solutions:

# Fix 1: Change indentation level of the print statement.
def func():
    t = 35
    print(t)

# Fix 2: Save 't' in another namespace.
def func():
    t = 35
    return t

t = func()
print(t)


I am working on the file "A Beginners Guide to Python/22. Common Error Messages.ipynb".
The first task is:
One fixed oversight later, the code works.
Index Errors
Index Errors occur when you are trying to index into an object but the index value you have chosen is outside the accepted range. As a quick recap, the accepted range is -length to length -1. For example, if my list has 10 items then I'll receive an index error if I try hand in a number outside the range -10 to 9.
Can you write Python code for it?

lst = [0] * 10

print(lst[-10]) # Works!
print(lst[-11]) # Fails


I am working on the file "A Beginners Guide to Python/22. Common Error Messages.ipynb".
The first task is:
So what this code is trying to do is simple; the user enters two numbers(A, B) we add them up and then return True if the result is a perfect square.
However, the code didn't work, and the trace-back is flagging an error with the ‘is_square’ function. In truth however, the problem happened much further back in the code, the is_square function is not our problem.
Arguably we have a problem with our addition function, for although it works, it also, thanks to operator overloading, works on strings and numbers alike. Instead of receiving an error at this juncture we instead send 'junk' to the is_square function. For example, if a and b are 10 and 6 then add(a, b) should return integer 16, not string 106.
But we can go back even further in our analysis and ask:

"Why did our addition function receive strings as input in the first place?"

The actual source of this error is forgetting that the input function returns strings, and we did not convert those strings to integers. This error then trickled all the way through the rest of the program until we finally receive a type error far removed from the actual problem. The solution:
a = int(input(“{text...}”))

The moral of the story here is that when you receive type errors the source of the problem often isn't the bit of code that raised the error. I’d recommend writing print statements at different parts of the code to see if everything is giving the correct output.
Misunderstanding Properties...
Another source of Type errors occurs when you don't fully understand the properties of a particular data-type. Or maybe you misunderstand how a particular object method works. Or maybe you err because you don't understand how something is implemented within Python (at the low-level). The usual remedy for this sort of error is documentation and/or google.
Here's a simple example:
Can you write Python code for it?

a_list = [1,2,3,4]
print(a_list[-1])  # so far so good.
a_list[-1] = 99    # Seems legit.
print(a_list)

# And now with strings...

a_string = "abcde"
print(a_string[-1]) # still working...
a_string[-1] = "zztop" # Oh noes! a Type Error
print(a_list)


I am working on the file "A Beginners Guide to Python/22. Common Error Messages.ipynb".
The first task is:
The code above intends to take a list and for each item multiply that item by itself, [1,2,3] ---> [1, 4, 9]. Although probably not the best approach this code is generally correct but for an easy to fix oversight; we can't call range on a list! What we actually meant to do was call range on the length of the list. Like so:
Can you write Python code for it?

l = [1,2,3]

for n in range(len(l)):
    l[n] *= l[n]
print(l)


I am working on the file "A Beginners Guide to Python/22. Common Error Messages.ipynb".
The first task is:
Type Errors
Suppose you have an object of Type ‘X’ (Int, str, ...) and some sort of operation ‘Y’ (multiplication, concatenation). The Type error happens when operation ‘Y’ is not compatible with object Type ‘X’.
For example: A / B makes sense when A and B are floats/integers, but Python does not know what it is to divide a list by a list nor does it understand what you want to do when you try to divide the string "cat" with the set '{1, 2, 3}' or something. In such cases you get a type error.
Can you write Python code for it?

10 / 2        # works!
"abc" / "de"  # error!


I am working on the file "A Beginners Guide to Python/22. Common Error Messages.ipynb".
The first task is:
Typo's
Typo's are probably the main cause of syntax errors, check that you haven't missed things like brackets, colons, comma's and so on. A few examples...
Can you write Python code for it?

lst = [1 2 3 4] # No comma's between items. Fix is: lst = [1, 2, 3, 4]
10 + 12) * (4 + 3) # missing brackets. Fix is: (10 + 12) * (4 + 3)


I am working on the file "A Beginners Guide to Python/23. Try & Except.ipynb".
The first task is:
In this case Python didn't receive a TypeError and thus the except block of code failed to execute. Now, we can fix this code in one of two ways:
Can you write Python code for it?

x = 0

# The bad fix first...
try:
    x / x
except:
    print("Bad ", x + x)

# The Good fix...
try:
    item / item
except (TypeError, ZeroDivisionError): # please note the "SnakeCase".
    print("Good", x + x)


I am working on the file "A Beginners Guide to Python/23. Try & Except.ipynb".
The first task is:
So what’s going on here? Well, we have list which contains several different data-types. For every 'item' we try to multiply item by itself. If 'item' is a number this makes sense and so we print item * item. However, if we try to multiply a string by a string we get a TypeError, which the except statement catches. So if we receive a TypeError we try something else, in this particular case we add item to item, and thus "aa", "bb", etc get printed.
Now it is important to note this current code is only set up to handle TypeErrors. What happens if we change the above bit of code to divide rather than multiply and feed it 0?
Can you write Python code for it?

item = 0
try:
    item / item
except TypeError:
    print(item + item)


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
And now its not really possible to make the test any simpler. But by now the problem should be easy to understand; it seems as if the first number we are looking for is 2, so if 1 is missing we fail.
Lets test that hypothesis by writing more tests!
Can you write Python code for it?

print_test_result(firstMissingPositive, [1,2,3], 4)
print_test_result(firstMissingPositive, [1, 3], 2)
print_test_result(firstMissingPositive, [1], 2)


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
Most of the time however, the code you want to test will not be so easily reversed engineered. So most of the time your tests are going to be hand-written. And because writing tests can be a bit tedious and time consuming you are going to want tests that are:

Fast to write and execute
Easy to understand, they should give clear feedback on what went wrong
Test most/all of the likely scenarios.

For these reasons, its often a good idea to write tests that follow a common format. Great tests are often tests that you can copy and paste, and change into a new test by changing a small handful of values.
To illustrate that, lets suppose I have the following code:
Can you write Python code for it?

def firstMissingPositive(nums):
    """
    Given an unsorted integer array (nums) finds the smallest missing positive integer.

    for example:
    [0,1,2,3]  => returns 4, since 4 is the smallest missing number
    [10,11,12] => returns 1
    """
    return 3


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
Now, we know that X/0 is a ZeroDivisionError, the question is what do we want the result to be? Do we want Python to raise the error? or would we prefer Python to do something else such as return a number or perhaps a string.
Remember that errors are not bad, if Python to throws an error when it gets zero as input that’s totally fine, and in this case I think I’m happy with the error. This means I have to write a test case that expects an error to be raised. We can do that like so…
Can you write Python code for it?

try:
    divide(1, 0)
    print(False) # note that if the above line of code yields a zeroDiv error, this line of code is not executed.
except ZeroDivisionError:
    print(True) # Test pass, dividing by zero yields an error.


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
Okay so know that we have spent a little bit of time working on a test framework we can (a) quickly write new tests and (b) we can also clearly see why a test failed.
In test-driven development once you have a small selection of tests you then try to write code that passes the tests. Let's do that now...
Can you write Python code for it?

def firstMissingPositive(nums):
    """
    Given an unsorted integer array (nums) finds the smallest missing positive integer.

    for example:
    [0,1,2,3]  => returns 4, since 4 is the smallest missing number
    [10,11,12] => returns 1
    """
    i = 1
    while True:
        i += 1
        if i not in nums:
            return i


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
Okay so, this is where we need to put our ‘thinking hat’ on for a moment. The documentation for this function specifically states A and B are supposed to be numbers, so instead of wasting time breaking the code with obviously bad inputs (e.g strings) lets try to break with valid inputs. In other words:

what are the possible integers/floats we can pass in where this function may break?

When dealing with numbers there are, I think, three basic tests that are almost always worth running:

Negative Numbers
Zero
Positive Numbers

And in addition to those tests we should also run tests for:

Small inputs (e.g 10/5)
Very large inputs (e.g 999342493249234234234234 / 234234244353452424 )

You may remember for example in lecture 21 as we tried to optimise our is_prime function we introduced some defects when working with small numbers.
Anyway, the point is these five basic cases will cover a lot of situations you may have with numbers. Obviously you should run several tests for each of these basic test cases. And in addition to the basic tests you should run more function specific tests too; for example, if I have a function that returns the factors of n then it would be wise to run a bunch of tests with prime numbers to check what happens there. You should also test highly composite numbers too (e.g 720, 1260). In regard to our division function a good additional test would be when the numerator is larger than the denominator and vice versa (e.g. try both 10/2 and 2/10). Zero is also a special case for division, but we have already listed it in the basic tests.
Okay, so lets write our first tests:
Can you write Python code for it?


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
Okay we have simplified the test and it still fails. Lets make it even simpler!
Can you write Python code for it?

print_test_result(firstMissingPositive, [2], 1)
print_test_result(firstMissingPositive, [], 1)


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
Okay, next up we need to test for large numbers. When it came to small numbers we can easily work out the correct answer by hand, but for large sums that’s not so easy.
Your first instinct here might be to say "use a calculator" and while that’s true, that solution only works in this very specific case. What we actually want is a more general solution that can solve all sorts of problems.
It turns out that sometimes building code that can generate test cases is a lot easier that building the solver. In this particular example we can do just that...
Let's take a step back and ask ourselves what division actually is. The answer is basically the opposite of multiplication. And so, we can actually write test cases for our function by "reverse engineering" the problem. We know from math that the following is always true:
(y * y) / y = y
(x * y) / y = x

And so, so long as we have a function that multiplies correctly, we can be confident that our function is getting the right answer to complex division problems even though we do not know what the right answer is ourselves. In code:
Can you write Python code for it?


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
One test has failed, can you spot what the problem is?
When you have a failing test, and you are not sure exactly what the problem is a good thing to do is to try to make the test as simple as possible. Let's try that now.
Can you write Python code for it?


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
So we have some tests, if "False" gets printed that means the test failed. This is a good start. Notice how these tests are easy to write and understand. We can also quickly add tests by copy & paste plus some tweaks. On the downside, the output is not very informative; Why did a test fail? Here our only option is figure out what test failed and then re-run it, this time makeing a not of the value.
Can you write Python code for it?

print("Got:", firstMissingPositive([1,2,3]))
print("should be 4")


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
This is actually a hard problem to solve efficiently but I don't care about that. Right now, I only care about testing. And this is function that is easy to test.
Test-Driven Development
Sometimes, software developers write tests before they actually write the solution to thier problem. This is called "test-driven development". The advantage of writing tests first is that it forces you to think about the problem in a different way. Instead of thinking about how to solve the problem we instead start out by thinking about the sorts of inputs that are difficult. Sometimes, that means we spot problems faster than we would have otherwise.
Okay, lets write some tests!
Can you write Python code for it?

print(firstMissingPositive([1,2,3]) == 4)
print(firstMissingPositive([0,0,1]) == 2)
print(firstMissingPositive([1,2]) == 3)


I am working on the file "A Beginners Guide to Python/25. Introduction to Testing.ipynb".
The first task is:
We ran doctests, but since the test past nothing happened. Alright, lets show you want happens on failure:
Can you write Python code for it?

def run_all_the_tests():
    """
    >>> 1 + 1
    2

    >>> print(True)
    True

    >>> 20 + 2
    23
    """

    print("testing complete")

run_doctests()


I am working on the file "A Beginners Guide to Python/26. Design Decisions, How to Build Chess Game.ipynb".
The first task is:
At first glance this code seems pretty good, but there are a few drawbacks. Firstly it looks like we are going to be repeating ourselves a lot; queen movement for example is just a copy & paste of the rook + bishop. The king function is likely a copy & paste of queen but where we change the distance to 1.
And by the way guys, repeating oneself is NOT quite the same as reusing code!
What we would really like to do here is generalise the problem as much as we can. And good technique for doing that is to think of the next project we might want to implement. For example, let’s suppose after building my chess game I want to support Capablanca Chess ?
<img src="http://hgm.nubati.net/rules/Capablanca.png" style="width:200px;height:150px;" ALIGN="centre">
Capablanca chess is played on a 10x8 board and it has two new pieces; the ‘archbishop’ moves like a bishop combined with a knight and a ‘chancellor’ moves like a rook and a knight.
So, what should we do here? Well, I think the first thing we should do is define movement of pieces WITHOUT referencing a board. If we don’t reference a board that means we should be able to handle boards of many different sizes. Secondly, if we define pieces in terms of combining general patterns (e.g Queen = diagonal + orthogonal movement) then defining new pieces will probably be less the five lines of code in many cases.
Let’s examine what that might look like:
Can you write Python code for it?

# Note the following code doesn't work, it is for demonstration purposes only!

def diagonal_movement(position, directions, distance =1):
    """
    Returns all diagonal squares in all valid directions N distance from the origin
    """
    # code here
    pass

def othagonal_movement(position, directions, distance=1):
    """
    # doctests showing example useage:

    >>> othagonal_movement( (2, 2), directions=[left], distance= 2)
    [(2,2), (2, 1), (2, 0)]

    >>> othagonal_movement( (2, 2), directions=[right], distance= 4)
    [(2, 2), (2, 3), (2, 4), (2, 5), (2, 6)]

    >>> othagonal_movement( (2, 2), directions=[right, left], distance= 1)
    [(2, 1), (2,2), (2, 3)]
    """
    # code here
    pass


I am working on the file "A Beginners Guide to Python/26. Design Decisions, How to Build Chess Game.ipynb".
The first task is:
But lets take a step back for a moment, what are actually doing here?
When naming variables, a good practice is to state what the code does rather than state what it is used for. The reason this can be a code idea is that renaming a function/variable to something general can make code more reusable and modular. The process of thinking about a name may help you spot ideas and patterns that you may have otherwise missed. So, let me ask the question again, what are we actually doing here?
Suppose I give you a table of data, and I want you so sum up all the columns. For example:
[0, 1, 2]
[5, 6, 7]

returns [5, 7, 9]

How can we solve this problem? Well, notice that our othagonal_movement function can be used to solve this problem:
Can you write Python code for it?

def sum_column(table, column):
    num_rows = len(table[0])

    position = (0, column)
    points = othagonal_movement(position, direction=["down"], limit=num_rows)

    total = 0
    for p in points:
        x, y = p[0][1]
        number = table[x][y]
        total += number
    return total


I am working on the file "A Beginners Guide to Python/26. Design Decisions, How to Build Chess Game.ipynb".
The first task is:
Defining Piece movement
Alright, onto the next problem. "How are we going to make the peices move?" And once again a smart choice here will make things so much easier than it otherwise could be.
One simple approach is to write a function for each piece, like so:
Can you write Python code for it?

# Note the following code doesn't work, it is for demonstration purposes only!

def all_bishop_moves(position, board):
    """ When given a starting square, returns all squares the bishop can move to"""
    pass

def all_rook_moves(position, board):
    """ When given a starting square, returns all squares the rook can move to"""
    pass

def all_queen_moves(position, board):
    """When given a starting square, returns all squares the queen can move to"""
    pass


I am working on the file "A Beginners Guide to Python/26. Design Decisions, How to Build Chess Game.ipynb".
The first task is:
In the case of an 8x8 board 'limit' would be set to 8. If a piece is only allowed to move 1 square forward (regardless of board shape/size) we can easily model that by setting the limit to 1.
notice that with this design some peices can be defined very simply. For example, we can define a king in the following way:
Can you write Python code for it?

def king_movement(position, limit):
    return queen_movement(position, limit=1)


I am working on the file "A Beginners Guide to Python/26. Design Decisions, How to Build Chess Game.ipynb".
The first task is:
Okay so the board as a string seems possible, but are there any drawbacks of an implementation like this? Well, I can think of two. Firstly, notice that because these unicode characters are a bigger than normal letters we are going to need a new way to denote black and white squares. You can see from above I tried to use a combination of spaces and ‘□■’ characters but even then the formating is a bit off. In short, it looks like trying to get the board to look nice is going to be both tedious and fiddly.
What is the second problem?
You remember me mentioning that strings are an immutable data-type, which means that every time we want to change the board we have to make a new one. Not only would this be computationally inefficient it may also be a bit tricky to actually change the board.
For example, lets see what sort of work we would have to to make the move 1.Nf3:
Can you write Python code for it?

def make_move(move):
    """Takes a string a returns a new string with the specified move"""
    # Code here
    pass

# Our new function would have to take the original string and return the new string (both below)...
original_string = "RNBQKBNR\nPPPPPPPP\n-x-x-x-x\nx-x-x-x-\n-x-x-x-x\npppppppp\nrnbkqbnr"
new_string =      "RNBQKBNR\nPPPPPPPP\n-x-x-x-x\nx-x-x-x-\n-x-x-n-x\npppppppp\nrnbkqb-r"
print(new_string)


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_01. Building the Board (HW).ipynb".
The first task is:
Explaining my code
Okay guys, my code is a bit complicated to understand at first glance, but I'm going to talk you through it.
board_temp = ["B"] * bomb_count + [non_bomb_character] * (row * col-bomb_count)

So this line of code looks like a monster, but actually once you think about it is not so difficult to understand. Basically we start by making a list containing "B" characters, the number of "B"'s we add to the list is dependant on the number of bombs the board is supposed to have (which is given by the bomb count argument).
Can you write Python code for it?


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_01. Building the Board (HW).ipynb".
The first task is:
Possible Solution
Can you write Python code for it?

import random

def build_board(num_rows, num_cols, bomb_count=0, non_bomb_character="-"):
    board_temp = ["B"] * bomb_count + [non_bomb_character] * (num_rows * num_cols - bomb_count)

    if bomb_count:
        random.shuffle(board_temp)

    board = []
    for i in range(0, num_rows*num_cols, num_cols):
        board.append(board_temp[i:i+num_cols])
    return board

# Runing the tests...
test_board()
# Note if you recieve an error message saying test_board not found
# try hitting the run button on the test_board cell and try again.


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_01. Building the Board (HW).ipynb".
The first task is:
The next step is to multiply the row by the col to get the total number of squares our board should have. We subtract the bomb_count from this figure because we have already added those to the board and we do not wish to 'double count'.
For example, if bomb_count is 10 and our board is 10x10 then we want to add 90 non-bomb characters to the list (10*10 - 10 = 90). The result is a list of length 100, 10 of which are bombs.
Can you write Python code for it?

row = 2
col = 2
b_count = 1

part_one_of_list = ["B"] * b_count
part_two_of_list = ["-"] * (row * col-b_count)

our_list = part_one_of_list + part_two_of_list

print(our_list)  # --> 2x2 grid with 1 bomb.


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_01. Building the Board (HW).ipynb".
The first task is:
The next two lines of code are:
    if bomb_count:
        random.shuffle(board_temp)

So this code shuffles the list (but only if bombs >= 1), we take a list with all the bombs at the start and we jumble them around.
 board = []
 for i in range(0, row*col, col ):
    board.append(board_temp[i : i+col])

So this is the most complex part of my function, what are we doing here?
We have a range function, which starts at 0, ends at row * col and has a step of size col. Next up, we use those values to slice the temp_board we made earlier. In effect this is creating a 'moving window', and this gives us our board with the correct dimensions.
Lets try to visualise what is happening here by running it a few times with print statement:
Can you write Python code for it?

## Create a 1d board
board_temp = [str(i).zfill(2) for i in range(1,21)]
print(board_temp)

row = 5
col = 4

# print board with dims (5, 4)
for i in range(0, row*col, col ):
    print(board_temp[i : i+col])

row = 3
col = 9

# print board with dims (3, 9)
for i in range(0, row*col, col ):
    print(board_temp[i : i+col])

row = 6
col = 2

# print board with dims (6, 2)
for i in range(0, row*col, col ):
    print(board_temp[i : i+col])


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_02. Printing the Board(HW).ipynb".
The first task is:
Alternate Solution
Can you write Python code for it?

def display_board(board):
    print(*board, sep="\n")

# Runing the tests...
test()
# Note if you recieve an error message saying test_board not found
# try hitting the run button on the test_board cell and try again.


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_02. Printing the Board(HW).ipynb".
The first task is:
Possible Solution
Can you write Python code for it?

def display_board(board):
    for row in board:
        print(row)

# Runing the tests...
test()
# Note if you recieve an error message saying test_board not found
# try hitting the run button on the test_board cell and try again.


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_04. Getting the Neighbours(HW).ipynb".
The first task is:
Explanation
Neighbour Squares
'neighbour_squares' takes an (x, y) pair co-ordinate and returns the neighbours of that square. Often a square has eight neighbors (up left, up right, below, below right, etc), however squares on the edge of the board have fewer. The purpose of "row_check" and "col_check" is to help avoid possible index errors later on and also avoids a possible bug where the index is negative. For example:
[1,2,3]
[4,5,6]
[7,8,9]

square(3) has an index of (0,2). If we subtract 1 from both sides we get (-1,1). And the index (-1,1) is equivalent to (2,1), thus we end saying the neighbour of square(3) is square(8). This is only desirable in the cases where we want the edges of the board to 'wrap round'.
Carving this logic out into its own function (as opposed to doing everything in the count_bombs function) allows us to be flexible and this code can be reused (notice that "neighbour_squares" doesn't even take a board as an argument).
Count Bombs
The 'count_bombs' function takes an (x,y) position and a board. If then returns how many neighbour squares are in fact bombs. This function can however be improved.
You may remember me saying in the design lecture that its usually better to give a function a name that describes what it does and not what it is used for. This functions name ("count_bombs") makes this mistake. To fix this problem, we have to ask ourselves:
"What this function is actually doing?"

Here's what it is not doing: counting_bombs. A bomb in our current implementation is simply the string "B". So what this function is actually doing is counting how many occurrences of the string "B" there are. That's not a very useful function. A much more useful function would be to count any arbitrary character. So let's change the function name and signature to better reflect this new understanding
def count_occurence_of_character_in_neighbour_squares(x, y, array, character):
    ...

And thats a function that we could use easily in a chess game:
ENEMY_PIECE  = "P"

chess_board = ["_, "P", _]
...
Can you write Python code for it?

def get_square(x, y, board):
    """
    This function takes a board and returns the value at that square(x,y).
    """
    return board[x][y]

def count_occurence_of_character_in_neighbour_squares(x, y, board, character):
    """
    returns the number of neighbours of (x,y) that are bombs. Max is 8, min is 0.
    """
    num_rows = len(board[0])
    num_cols = len(board)

    squares = neighbour_squares(x, y, num_rows, num_cols)

    character_found = 0
    for px, py in squares:

        square_value = get_square(px, py, board)

        if square_value == character:
            character_found += 1

    return character_found


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_04. Getting the Neighbours(HW).ipynb".
The first task is:
Possible Solution
Can you write Python code for it?

def neighbour_squares(x, y, num_rows, num_cols):
    """
    (x, y) 0-based index co-ordinate pair.
    num_rows, num_cols: specifiy the max size of the board

    returns all valid (x, y) coordinates from starting position.
    """
    offsets = [(-1,-1), (-1,0), (-1,1),
               ( 0,-1),         ( 0,1),
               ( 1,-1), ( 1,0), ( 1,1)]

    result = []
    for x2, y2 in offsets:

        px = x + x2
        py = y + y2

        row_check = 0 <= px < num_rows
        col_check = 0 <= py < num_cols

        if row_check and col_check:
            point = (py, px)
            result.append(point)

    return result

def count_bombs(x, y, board):
    """
    returns the number of neighbours of (x,y) that are bombs. Max is 8, min is 0.
    """
    num_rows = len(board[0])
    num_cols = len(board)

    squares = neighbour_squares(x, y, num_rows, num_cols)

    bombs_found = 0
    for px, py in squares:
        if board[px][py] == "B":
            bombs_found += 1

    return bombs_found

# Testing...
test_neighbour_squares()
test_count_bombs()


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_06. Revealing Squares(HW).ipynb".
The first task is:
FUCK!
Notice that the player_board reveals a 1 here. But there is clearly two bombs nearby. What this means is that I have a bug somewhere in my code.
It could be that my game_boards and player_boards are out of sync, or (more likely) I have some sort of indexing error in the "count_occurence_of_character_in_neighbour_squares" function. My best guess is that I've mixed up (x,y) coordinates somewhere.
But, in a way, this is a good thing. It's a good thing because it gives me an excuse to talk a bit more about testing and debugging code. In development there is a concept called "failing fast". At heart it's a simple idea; you should test your code quickly and often. And you should do that because it's much better to see code fail when it is small and simple than when its large and complex.  If you have two thousand lines of code then that means there are two thousand possible places where the bug could be. If there are only twenty lines of code then there are only twenty possible places where the bug could be.
The other advantage of testing code as soon as possible is that the longer bugs go unnoticed the harder they are to fix.

Okay I've made a mental note of the error, but let's carry on for the moment. Maybe we shall find another bug...
Can you write Python code for it?

## Square already revealed, nothing happens...
reveal_square(1, 0, player_board, game_board)

print("INTERNAL GAME STATE:")
display_board(game_board)
print("")
print("PLAYER BOARD:")
display_board(player_board)

## Flag a square
flag_square(2, 0, player_board)

print("INTERNAL GAME STATE:")
display_board(game_board)
print("")
print("PLAYER BOARD:")
display_board(player_board)

## Deflag square
flag_square(2, 0, player_board)

print("INTERNAL GAME STATE:")
display_board(game_board)
print("")
print("PLAYER BOARD:")
display_board(player_board)

## Reveal a bomb, Should display game over
reveal_square(1, 1, player_board, game_board)

print("INTERNAL GAME STATE:")
display_board(game_board)
print("")
print("PLAYER BOARD:")
display_board(player_board)


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_06. Revealing Squares(HW).ipynb".
The first task is:
Possible Solution:
Can you write Python code for it?

def flag_square(row, col, player_board):
    p_square = get_square(row, col, player_board)

    if p_square in "012345678":
        # do nothing
        return

    ## set flag
    if p_square == EMPTY_SQUARE_CHARACTER:
        set_square(row, col, FLAG_CHARACTER, player_board)

    ## Deflag if flag is already set
    if p_square == FLAG_CHARACTER:
        set_square(row, col, EMPTY_SQUARE_CHARACTER, player_board)
        return

def reveal_square(row, col, player_board, game_board):
    p_square = get_square(row, col, player_board)

    if p_square in "012345678" or p_square == FLAG_CHARACTER:
        ## do nothing
        return

    g_square = get_square(row, col, game_board)

    if g_square == BOMB_CHARACTER:
        return game_over()

    else:
        bomb_count = count_occurence_of_character_in_neighbour_squares(row, col, game_board, BOMB_CHARACTER)
        set_square(row, col, str(bomb_count), player_board)

def game_over():
    print("GAME OVER")
    ## Later on, we can implement more logic here, such as asking if the player wants to play again.


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_06. Revealing Squares(HW).ipynb".
The first task is:
The above code should be familiar to you. The only addition I've made here is that I've added some character definitions. I thought it was good idea to define what the bomb_character is just once and have all the other functions reference that constant value.
When you start to glue the various peices of code together its perfectly normal to come up with new (and often better) ideas. So this means I'll need to go back and 'refactor' some of my older code to utilise this.
Anyway, how about we play a game?
Can you write Python code for it?

print("INTERNAL GAME STATE:")
display_board(game_board)
print("")
print("PLAYER BOARD:")
display_board(player_board)

## Player board should display 2.
reveal_square(1, 0, player_board, game_board)

print("INTERNAL GAME STATE:")
display_board(game_board)
print("")
print("PLAYER BOARD:")
display_board(player_board)


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_07. My Solution (explanation).ipynb".
The first task is:
By the way, in python it is possible to save a function as a variable (with the idea of calling it later), which the code below below hopefully illustrates:
Can you write Python code for it?

def example(number):
    return number

m = example # example is NOT called. m is merely a reference to a function.
n = example(20) # n calls example with the argument 20. The result is a number

print(n)
print(m(20)) # m(20) is the same as example(20)


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_07. My Solution (explanation).ipynb".
The first task is:
Combining all of these things means it is possible to write an argument parser in just a few lines of code. Moreover, adding new commands requires very little effort.
another reason why my implementation is more powerful than if statements is that I can, with minor modifications use it in other projects. For example:
Can you write Python code for it?

def parse_command(command, command_dictionary):
    """
    command: str
    command_dictionary: dict where the key a command and the value is a function reference
    """
    instruction, *arguments = command.split(" ")

    if instruction in command_dictionary:
        return command_dictionary[instruction](*arguments)
    else:
        return f"ERROR: '{instruction}' is not a valid command"

math_dict = { "sqrt": lambda x: int(x)**0.5,
              "round": lambda x, precision: round(float(x), int(precision)),
              "neg": lambda x: -float(x) }

string_dict = { "toCaps": str.upper,
                "reverse": lambda x: x[::-1],
                "join": lambda *x: "".join(list(x))}

print("STRING_DICT EXAMPLES...")
print(parse_command("toCaps hello", string_dict))
print(parse_command("reverse dlrow", string_dict))
print(parse_command("join h e l l o _ w o r l d", string_dict))
print()

print("MATH_DICT EXAMPLES...")
print(parse_command("sqrt 2", math_dict))
print(parse_command("round 10.98 1", math_dict))
print(parse_command("neg -2", math_dict))

print(parse_command("missing a b c", math_dict))


I am working on the file "A Beginners Guide to Python/Final Project (Minesweeper)/_07. My Solution (explanation).ipynb".
The first task is:
So this code would more or less do the same job. And although its easier to understand it does have the drawback that for every additional command we add we need to add several lines of code. Meanwhile, the solution I went for does away with all those nested if statements. In fact, adding an extra command requires just a single line of code (which we add to the COMMANDS dictionary).
The 'cleverness' of my implementation can be seen in these two lines:
instruction, *arguments = command.split(" ")
COMMANDS[instruction][0](*arguments)

The first line is more or less equivlent to the following:
instruction = command[0]
arguments   = command[1:]

The second line can only be understood with reference to the COMMANDS dictionary. Basically every command has a tuple with two elements. The first element (index 0) is the function we want to call (index 1 is the 'help text'). Meanwhile *arguments will take a list of items and pass them to the function individually.
Can you write Python code for it?

def add(a, b):
    return a + b

nums = [1, 2]

# add(nums) # this would fail
print(add(nums[0], nums[1]))
print(add(*nums))


I am working on the file "A Beginners Guide to Python/Homework Solutions/06. Names (HW).ipynb".
The first task is:
Even Better Solution
Can you write Python code for it?

PI = 3.14
diameter = 2

circumference = PI * diameter


I am working on the file "A Beginners Guide to Python/Homework Solutions/06. Names (HW).ipynb".
The first task is:
Example Solution
Can you write Python code for it?

# calculate the circumference of a circle
pi = 3.14
diameter = 2

circumference = pi * diameter


I am working on the file "A Beginners Guide to Python/Homework Solutions/13. Strings & Methods (HW 2).ipynb".
The first task is:
Possible solution #1
Here we sort by language, and for each language the greeting is listed before goodbye message. Empty lines are used to separate the languages. All variable names are in the format:
{language}_{greeting/goodbye}
Can you write Python code for it?

spanish_greeting = "hola"
spanish_goodbye = "buenas noches"

japanese_greeting = "konichiwa"
japanese_goodbye = "sayonara"

english_greeting = "hello"
english_goodbye = "sod off, lad"

french_greeting = "bonjour"


I am working on the file "A Beginners Guide to Python/Homework Solutions/24. Getting Help (HW).ipynb".
The first task is:
So basically the issue is was that when we write [2]*3 and try to add it to a list python isn't making three separate lists, rather, its adding the same list three times!
The fix then, we need to make sure Python knows we want three separate lists, which we can do with a for loop:
Can you write Python code for it?

x = []
for i in range(3):
    x.append([2]*3)

print(*x, sep="\n")

print()

x[0][0] = "ZZ"
print(*x, sep="\n")


I am working on the file "A Beginners Guide to Python/Homework Solutions/24. Getting Help (HW).ipynb".
The first task is:
To see what's happening lets rewrite the code to make the issue even clearer:
Can you write Python code for it?


I am working on the file "A Beginners Guide to Python/Homework Solutions/24. Getting Help (HW).ipynb".
The first task is:
What I wanted to do was build a nested list, x is supposed to look like:
[[2, 2, 2],
 [2, 2, 2],
 [2, 2, 2]]

And then I wanted to change the value at index [0][0] but notice that instead of a single value changing the first item in every list changes. I wanted:
[["ZZ", 2, 2],
 [  2,  2, 2],
 [  2,  2, 2]]

But I got:
[["ZZ", 2, 2],
 ["ZZ", 2, 2],
 ["ZZ", 2, 2]]

Wierd right?
Your homework for this week it to search google for an awnser to this problem. Why doesn't X behave like I want it too and what to I need to make it work?
I know for a fact the awnser is on stackoverflow already (and probably hundreds of other websites too), so this is a test of your googling skills:
What search query is likely to return the information you need?

This excerise is a really useful I think. Through-out your programming carrear you are going to stumped on tough questions. In many cases, the fastest way to solve your issue is going to be google search. BUT to get the most out of a search engine is going to require you to carefully think about your problem and what query might contain the awnser.
Possible Solution
Google to the rescue! Okay so let's think about this problem a little bit; what "buzz words” might we need to feed google in order to get the answer we want?
Lets try...

Python

Err...yeah, I think we are going to need to be a bit more specific than that.

Python nested list

This search is a bit better, I mean, from the context alone Google has probably figured out that we are not interested in snakes! But again, still probably not specific enough.

Python nested list sublist assignment

This query seems decent right? It seems pretty descriptive of the problem, afterall.
Lets run it!
Google Search (15th June 2017)

The third hit sounds promising, lets go there and see.
...And sure enough, it sounds like someone is dealing with the EXACT ISSUE we had. The top-voted reply not only explains the issue but also the fix.
Basically, the issue is that when you write [ [0]*3] *3 ] we are actually storing a reference to...
Can you write Python code for it?

out=[[0]*3]*3
print( id(out[0]) )
print( id(out[1]) ) # want to know what "id" is? Why not read the documentation!


I am working on the file "A First Class.ipynb".
The first task is:
Even without knowing what a matrix is, or how to multiply them, the idea the a pair of objects might be inverses of one another, such that their product is 1 (one), is familiar, from ordinary number types.  2 and 1/2 are inverses in this sense, because 2 * (1/2) equals 1.
In this chapter, we'll create a template for a type of object called a Permutation, or P for short.  P-objects will have inverses and p * ~p (p times inverse p) will give the Identity permutation.
What would be the identity permutation?  A permutation that maps every element to itself, such as 'a' to 'a', 'b' to 'b' and so on.  Encrypting a message with the identity cipher would not change the message one bit.
We're going to want our permutations to multiply apparently, and take the unary operator ~ for "invert", such that p * ~p turns out to be legal Python.  Just as we might multiply matrices together, as above, we'd like to multiply permutations.  How might this be arranged?
Below is a lot more code in a single cell than we've yet seen.  We're defining the blueprint for a P-type object, a Permutation.  The objects made from this template are the "selves" i.e. each has a unique "self" that it brings to the table, when a method is run.
p.shuffle( ) actually triggers P.shuffle(p) behind the scenes, meaning p gets to be its own first argument to P.shuffle, which is why we write self as the first parameter. The interpreter itself supplies that argument, knowing p has one, or is one (a self).
Instances all share the same methods, but as selves each has room for it's own data, distinct from that of the other selves.  We plan to make this clear.
Think of how many permuations one might build from the 27 elements we've been using.  The letter 'a' could map to 27 other elements, including itself.  Every possible ordering of a-z plus space is fair game.  That's 27 factorial, a huge number.
Can you write Python code for it?


I am working on the file "A First Class.ipynb".
The first task is:
Just going by the names of the methods, which look like functions, indented under the keyword class, we see ideas developed in chapter 2, around encrypting and decrypting.  The main difference is the bare dictionary form of our permutation will be named _code and used internally.  We're then adding capabilities that bare dictionaries do not have, such as the ability to multiply, or invert thanks to a single operator.
Can you write Python code for it?

p = P()
p

p = p.shuffle()
p

p.encrypt('able was i ere i say elba')

p.decrypt(_)  # reuse the most recent result

q = ~p  # get the inverse
I = q * p

I # is it true?  Is I the identity permutation?

I.encrypt("this will do nothing hooray for identity permutation")


I am working on the file "A First Class.ipynb".
The first task is:
Lets start though, by completing our function from the last chapter.  We'll morph it into the method from_cyclic and have it it build us a new P-object.  Lets do that by refactoring, having the P object inherit from another class:
Can you write Python code for it?

class P_base:
    """
    A Permutation

    self._code: a dict, is a mapping of iterable elements
    to themselves in any order.
    """

    def __init__(self, iterable = ascii_lowercase + ' '): # default domain
        """
        start out with Identity
        """
        try:
            seq1 = iter(iterable)
            seq2 = iter(iterable)
        except:
            raise TypeError

        self._code = dict(zip(seq1, seq2))

    def __getitem__(self, key):
        return self._code.get(key, None)

    def __repr__(self):
        return "P class: " + str(tuple(self._code.items())[:3]) + "..."

    def __mul__(self, other):
        """
        look up my keys to get values that serve
        as keys to get others "target" values
        """
        new_code = {}
        for c in self._code:  # going through my keys
            target = other._code[ self._code[c] ]
            new_code[c] = target
        new_P = P()
        new_P._code = new_code
        return new_P

    def __eq__(self, other):
        """
        are these permutation the same?
        Yes if self._cod==e  other._code
        """
        return self._code == other._code

class P(P_base):  # first use of inheritance

    def __invert__(self):
        """
        create new P with reversed dict
        """
        newP = P()
        newP._code = dict(zip(self._code.values(), self._code.keys()))
        return newP

    def shuffle(self):
        """
        return a random permutation of this permutation
        """
        # use shuffle
        # something like
        the_keys = list(self._code.keys()) # grab keys
        shuffle(the_keys)  # shuffles 'em
        newP = P()
        # inject dict directly...
        newP._code = dict(zip(self._code.keys(), the_keys))
        return newP

    def encrypt(self, plain):
        """
        turn plaintext into cyphertext using self._code
        """
        output = ""  # empty string
        for c in plain:
            output =...


I am working on the file "A First Class.ipynb".
The first task is:
Multiplication is not Commutative in this case.
Lets take a look at pq as a tuple of cyclic subgroups.  We call them subgroups because each tuple describes a permutation with the Group properties above.
Can you write Python code for it?


I am working on the file "A First Class.ipynb".
The first task is:
Properties of a Group
So you see the matrix type (numpy.matrix) and the permuation type (P) have some similarities. In Group Theory, a branch of mathematics, we talk about the properties of a group, a set of objects and an operation, usually called multiplication.
These properties are:

Closure (two elements multiplied always give another element in the set)
Associativity (p * q) * r == p * (q * r)
Inverses (such that p * ~p is the neutral element)
Neutral Element (Identity)

Some groups are also Abelian, meaning their operation is also Commutative, not just Associative.
These concepts might seem foreign to learning a computer language such as Python, but Closure should remind you of types, and how two objects of the same type may or may not result in a new object of that type.
For example two integers always add to give an integer, but the multiplicative inverse of an integer is not an integer except in the case of 1 itself.  2 * (1/2) equals 1.  1/2 is the inverse of 2.  The set of integers is not a group with respect to multiplication, as one property of a Group is every object has its inverse.
So what does it mean to "multiply" two permutations anyway?  If the first one maps 'a' to 'q' and the second maps 'q' to 'k', then their product permutation takes us right from 'a' to 'k'. You could think of multiplying as a kind of "chaining" or "pipelining" in this case.
Can you write Python code for it?

p = P().shuffle() # quick way to start out randomized
q = P().shuffle() # another one
interim  = p['a'] # a goes to something
interim

terminus = q[interim] # something goes to final stop
terminus

pq = p * q
terminus == pq['a']    # should be true

p * q == q * p


I am working on the file "A Look at Bicycle Commuting in the US – Part I.ipynb".
The first task is:
1. Data Wrangling
You can download a fresh version of the data file from Data.gov, or just download the one I uploaded to the repo.*
* Note: The file in the repo has the state abbreviations already added under the column called 'Code'.
Can you write Python code for it?

### Reading the data and assessing data frame structure
df = pd.read_csv('BikeState.csv', header=0)
df.info()

### Data Cleaning and Preprocessing
# Dealing with missing values
df = df.replace(' ', np.nan)
df = df.dropna()
df = df.reset_index()

# Making column titles a bit more manageable
df['funds_raw'] = df['Use of Federal Funds for Bicycle and Pedestrian Efforts: Raw Value'].astype(float)
df['funds'] = df['Use of Federal Funds for Bicycle and Pedestrian Efforts: Score'].astype(int)
df['bike_score'] = df['Commute Mode Share - Bicycle: Score'].astype(int)
df['bike_share'] = df['Commute Mode Share - Bicycle: Raw Value'].astype(float)
df['physical_score'] = df['Physical Activity from Transportation: Score'].astype(float)
df['physical_raw'] = df['Physical Activity from Transportation: Raw Value'].astype(float)
df['bike_fatalities_score'] = df['Road Traffic Fatalities per 100,000 Residents - Bicycle: Score'].astype(int)
df['bike_fatalities_exposure'] = df['Road Traffic Fatalities Exposure Rate - Bicycle: Raw Value'].astype(float)
df['walk_score'] = df['Commute Mode Share - Walk: Score'].astype(int)
df['walk_share'] = df['Commute Mode Share - Walk: Raw Value'].astype(float)
df['transit_score'] = df['Commute Mode Share - Transit: Score'].astype(int)
df['transit_share'] = df['Commute Mode Share - Transit: Raw Value'].astype(float)
df['auto_score'] = df['Commute Mode Share - Auto: Score'].astype(int)
df['auto_share'] = df['Commute Mode Share - Auto: Raw Value'].astype(float)
df['proxmajorhwy_score'] = df['Proximity to Major Roadways: Score'].astype(int)
df['proxmajorhwy_raw'] = df['Proximity to Major Roadways: Raw Value'].astype(float)
df['mileswalk_score'] = df['Person Miles of Travel by Walking: Score'].astype(int)
df['mileswalk_raw'] = df['Person Miles of Travel by Walking: Raw Value'].astype(float)
df['duidwi_score'] = df['DUI/DWI Fatalities per 10,000 Residents: Score'].astype(int)
df['duidwi_raw'] = df['DUI/DWI Fatalities per 10,000 Residents: Raw Value'].astype(float)

# Transforming categorical...


I am working on the file "A Look at Bicycle Commuting in the US – Part I.ipynb".
The first task is:
Plot VI: Line of Best Fit – Physical Activity and Commuting by Public Transit
Can you write Python code for it?

x = df['transit_share'].astype(float)
y = df['physical_raw'].astype(float)
ax = sns.regplot(x=x, y=y, data=df, color=p[4])
ax.set_title('Line of Best Fit – Physical Activity and Commuting by Public Transit')
ax.set(xlabel='Public Transit Commute Share', ylabel='Physical Activity Raw Value')
plt.show()


I am working on the file "A Look at Bicycle Commuting in the US – Part I.ipynb".
The first task is:
The regression plots above indicate that biking, walking, and using public transit are loosely positively correlated with physical activity, and commuting by car is loosely negatively correlated with physical activity.
Plot VIII: Histogram - Complete Streets Policies
I'm also interested in seeing how Complete Streets Policies affect physical activity across the US. States and regions either enact those policies or they don't, and it's possible that those that do observe higher levels of alternative transportation.
Can you write Python code for it?

x = df['CSP']
tr1 = Histogram(x=x, histnorm='probability',
                xbins=dict(start=np.min(x), size= .99, end= np.max(x)),
                marker=dict(color='rgb(200, 200, 200)'))

title = "Historgram<br>Complete Streets Policy (Bernoulli)"

layout = dict(
            title=title,
            autosize= True,
            bargap= 0.01,
            height= 700,
            width= 900,
            hovermode= 'x',
            xaxis=dict(
            autorange= True,
            zeroline= False),
            yaxis= dict(
            autorange= True,
            showticklabels= True,
           ))
fig = Figure(data=Data([tr1]), layout=layout)
iplot(fig, filename='Histogram Complete Streets Policy (Bernoulli)', validate=False)


I am working on the file "A Look at Bicycle Commuting in the US – Part I.ipynb".
The first task is:
Walk values are noticeably skewed to the right, and bike values resemble a Bernoulli distribution.
Plots IV: Line of Best Fit – Physical Activity and Commuting by Bike
Now that we have an idea of the probability distributions for our variables of interest, it's time to build a few simple linear regression plots for pattern recognition–to see if we can identify any correlative relationships between physical activity and the commute covariates.
Can you write Python code for it?

x = df['bike_share'].astype(float)
y = df['physical_raw'].astype(float)
ax = sns.regplot(x=x, y=y, data=df, color=p[5])
ax.set_title('Line of Best Fit – Physical Activity and Commuting by Bike')
ax.set(xlabel='Bike Commute Share', ylabel='Physical Activity Raw Value')
plt.show()


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
A few more practical uses of nparray:
Can you write Python code for it?

data.mean(), data.max(), data.min()

data[3].max()  # Max inflammation for 4th patient


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
A lot of the magic methods here, those with double underscores on either side, allow Python to treat this object more like a built-in. For instance, the __neg__ method will get called when we enter -data. If we try data1 < data2, then data1.__lt__ will be called with data2 as an argument, expecting a boolean return. The behaviour when the user writes data.previously_unknown_attr is defined by the __getattr__ method - this need not be an error. In some ways, you can think of this as an expansion of the concept of operator overloading.
The ndarray class uses those methods very effectively, to help us write clear and efficient code.
Can you write Python code for it?


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
As you can see, the second day does indeed have max of 1.0. Notice that it returns an array (this is a superclass of ndarray), so we can do...
An aside: this is the first time you have seen a named argument. In an argument list of a function, you can supply a name for arguments:
python
def some_function(arg1, info=38, axis=None):
   ...
To write a function like this, you must provide a default for the argument. You can then call the function only passing a subset of arguments, as you have just seen.
Can you write Python code for it?


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
Don't forget to put up your star!
Start with the Bokeh documentation and the Bokeh example above...
Can you write Python code for it?

# RMV
# This is the whole figure, possibly
# with multiple subplots
fig = bplot.figure()

days = np.arange(data.shape[1])

# This is a specific set of axes
fig.xaxis.axis_label = 'Day'
fig.yaxis.axis_label = 'Inflammation'

fig.line(days, data.mean(axis=0), legend='Average', color='green')
fig.line(days, data.max(axis=0), legend='Max', color='blue')
fig.line(days, data.min(axis=0), legend='Min', color='red')

bplot.show(fig)


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
First off, note that we do not have to tell the function to import the global variable, we can just use it. Now, examine the function - this is callback that we will use to update the plot when we change a drop-down menu choice of function.
We create a source object, an instance of the ColumnDataSource class, and when this function is called, it updates the 'y' item in source.data. Inside the function, we check what has been picked (this is the argument to the function), and update the source "y" array on that basis. Finally, we push this change to the notebook.
Can you write Python code for it?

fig = bplot.figure()

days = np.arange(data.shape[1])

fig.xaxis.axis_label = 'Day'
fig.yaxis.axis_label = 'Inflammation'

fig.line(initial_coordinates['x'], initial_coordinates['y'], source=source)
bplot.show(fig)


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
Interactive plots
But really, this still isn't very interactive... let's try adding a drop-down to pick which function (average, max, min) that we want to use.
Can you write Python code for it?

from bokeh.models import ColumnDataSource
from bokeh.io import push_notebook

# Start out with days vs average
initial_coordinates = {'x': days, 'y': data.mean(axis=0)}
source = ColumnDataSource(initial_coordinates)

# Define a callback to update the plot when we
# pick something else
def update_plot_statistic(statistic):
    if statistic == "Average":
        source.data['y'] = data.mean(axis=0)
    elif statistic == "Max":
        source.data['y'] = data.max(axis=0)
    elif statistic == "Min":
        source.data['y'] = data.min(axis=0)
    push_notebook()


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
MMMM. Pretty. Red shows greater inflammation, and what we may have guessed from the bits of CSV we saw, inflammation starts low, grows and reduces. Patients along the side, days along the bottom.
Time for a challenge... plot the heat map (as above) for the fourth, fifth and sixth patients, transposed with days up the side. The easiest way is to start with Google... But first, put up your stars so I can see them!
Can you write Python code for it?


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
Not particularly more complicated, but more flexible... say we have a series of plots...
Can you write Python code for it?

fig = pyplot.figure(figsize=(10.0, 3.0))

axes = []  # Blank list
for i in range(1, 4):
    ax = fig.add_subplot(1, 3, i)
    ax.set_xlabel('Day')
    axes.append(ax)

axes[0].set_ylabel('Average')
axes[1].set_ylabel('Max')
axes[2].set_ylabel('Min')

axes[0].plot(data.mean(axis=0))
axes[1].plot(data.max(axis=0))
axes[2].plot(data.min(axis=0))

fig.tight_layout()
pyplot.show(fig)


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
Notice the syntax is quite similar to matplotlib but a little different, so make sure you don't get them mixed up. For example, plot has become specifically line and we now explicitly provide the days along the bottom. Note that numpy.arange is just like range but it returns a numpy array.
Challenge II
Reproduce our three-line matplotlib plot in Bokeh
Can you write Python code for it?


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
Now for inline plotting we can use the matplotlib module, the go-to for Python plotting (but not in the Dijkstra sense)...
Can you write Python code for it?

import matplotlib
pretty_pic = matplotlib.pyplot.imshow(data)
matplotlib.pyplot.show(pretty_pic)


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
Now if I want to modify some part... I can do so and have the updated plot straight-away
This particular syntax is for fairly basic use, with only one set of axis and so forth, but it's probably quite familiar if you are coming from MATLAB. If we want to expand a bit more, first we use a more classic matplotlib approach, and neater programming approach, by setting properties directly on specific objects, such as axes...
Can you write Python code for it?

# This is the whole figure, possibly
# with multiple subplots
fig = pyplot.figure(figsize=(5.0, 3.0))

# This is a specific set of axes
axes = fig.add_subplot(1, 1, 1)
axes.set_xlabel('Day')
axes.set_ylabel('Inflammation')

axes.plot(data.mean(axis=0), label='Average')
axes.plot(data.max(axis=0), label='Max')
axes.plot(data.min(axis=0), label='Min')
axes.legend()

pyplot.show()


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
Reproducible Plots
If you come from a MATLAB background, or use other plotting tools, you will be used to saving/re-opening a plot...
...in Python you normally have a plotting script that you run each time (and save the image).
Cons

You have no standard GUI for designing plots
If your plot involves lots of processing, it's slow to open

Pros

Changes are trivially easy
Every time you open the plot, it contains current data

Con-workarounds

There are libraries for adding GUI elements to plots
You can split into two scripts
output processed data
plot

Example plotting script
Can you write Python code for it?

import numpy as np
from matplotlib import pyplot

data = np.loadtxt(fname='data/inflammation-01.csv', delimiter=',')
pyplot.figure(figsize=(5.0, 3.0))

pyplot.xlabel('Day')
pyplot.ylabel('Inflammation')

pyplot.plot(data.mean(axis=0), label='Average')
pyplot.plot(data.max(axis=0), label='Max')
pyplot.plot(data.min(axis=0), label='Min')
pyplot.legend()

pyplot.show()

exec(In[42])  # Cheeky way to reshow output... forget immediately.


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
This construct, a with statement, addresses the age-old problem of cleaning up file descriptors. In general, a with context expects the object being used to have some open and close routines that will be called at the entrance and exit of the block, respectively. Note that we don't have scoping here - the snippet variable exists outside the block, making it easy to load data in a short block and handle it later.
We could use more traditional syntax:
python
    f = open('foo.csv', 'r')
    ...
    f.close()
but this succinct approach has the benefit of containing use within a protected block.
It can be used for much more than just simple files, such as RMV.
We also note that a new piece of Python syntax has appeared - a star, indicating that an iterable (here, a list) is used to fill up arguments to a function. This is especially useful with print, and is equivalent to:
python
    print(snippet[0], snippet[1], snippet[2])
However, there are places it really shines. In fact, you can also name arguments to a function - in that case you can additionally provide a dict that fills them out (with two stars to indicate the expansion).
Looking at the printed data, you can see we have a large number of columns (equal for each row), data in the range 0-20 and it is comma-separated. Remember, each row represents a patient and each column, the measurement of inflammation on a given day.
Import the data
Can you write Python code for it?

import numpy as np
data = np.loadtxt(fname='data/inflammation-01.csv', delimiter=',') # Comma-separated...
print(data)


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
This shouldn't be surprising - we take the average over all patients for each of 40 days, so we end up with a 40 element array.
Try using the following cell to get the mean inflammation over all patients for each of the first three days...
(you should use slicing and end up with a 3 element array)
Can you write Python code for it?


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
This tells us that there are 60 patients (rows), each with 40 days of inflammation measurements (columns).
We can get an idea of the wide variety of methods available on the data object...
Can you write Python code for it?


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
This type is numpy's N-dimensional matrix class. This means we get a wide swathe of attributes and methods applicable to matrices, and a whole load of functions inside numpy and other libraries that will happily take it as an argument. One particularly useful property is shape:
Can you write Python code for it?


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
This works because numpy has overridden the __mul__ method.
We can also do our slicing in multiple dimensions... let's get the top-left corner...
Can you write Python code for it?


I am working on the file "A New Treatment for Arthritis.ipynb".
The first task is:
While this has kindly been rendered for us as if it were a list of lists, in reality it is something much more useful...
Can you write Python code for it?


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
As discussed earlier, the load_files function returns a data bunch which consists of {target_names,data,target, DESCR, filenames}. We can access a particular file from the training example as follows:
Can you write Python code for it?

# category of first document in the buch
print "TARGET NAME : " + training_data.target_names[training_data.target[0]]
# data of the first document in the bunch
print "DATA : " + training_data.data[0][:500]
# Target value of the first document in the bunch
print "TARGET : " + str(training_data.target[0])
# filename of the first document in the bunch
print "FILENAME: " + training_data.filenames[0]


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
Classification Report for LogisticRegression
Can you write Python code for it?

from sklearn.metrics import classification_report
print(classification_report(y_true, y_pred_logistic, target_names=training_data.target_names))


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
Classification Report for SGDClassifier
Can you write Python code for it?

from sklearn.metrics import classification_report
print(classification_report(y_true, y_pred_SGD, target_names=training_data.target_names))


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
Did you just say  64.392182842 % Accuracy?

The good news is we could do much better than this!!!!

The bad performance of Naive Bayes is a clear indication of the feature selection method that we chose. Hence in this section we will spend some time tweaking the features to see if we can improve the accuracy of the classifier.
Fortunately the CountVectorizer provides arguments that we can tweak. Let us dive right in.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
print count_vect


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
Fit SGDClassifier and LogisticRegression
Can you write Python code for it?

print ' [Classification Started: ' + str(datetime.datetime.now()) + ']'
# Fit the logistic regression and SGD
# For the logistic regression we will use the one vs rest classifier
clf1 = LogisticRegression(class_weight='auto',solver='newton-cg',multi_class='ovr').fit(X_train_tfidf, training_data.target)
clf2 = SGDClassifier(loss='log',class_weight='auto').fit(X_train_tfidf,training_data.target)
print ' [Classification ended: ' + str(datetime.datetime.now()) + ']'


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
From 64 to 67 % Accuracy

Now there isn't much tweak that you can do further to improve the accuracy of Naive Bayes. Though using alternate models such as Logistic Regression and Stochastic Gradient Descent may give better results, i want see if i can maybe improve the accuracy of the exisiting model by using some simple techniques.

 Instead of predicting one target label, we can tweak the program to maybe output the top 5 (say) predictions for every test document based on the probability value. This is quite easy to implement on the existing code and requires us to use the predict_proba function instead of the predict function in MultinomialNB. I'll show you in a second how to do this.
Can you write Python code for it?

from __future__ import division
#load test data
test_data= load_files('/media/arvindramesh/CCF02769F02758CA/TextClassification-NewsGroup/test/',shuffle=True, encoding='ISO-8859-2')
# variable to track the accuracy
count=0
print ' [Classification Started: ' + str(datetime.datetime.now()) + ']'
# Let's create a dictionary that stores the top 5 predictions for every test document
# Hence the key will be the document number and the value will be a list containing top 5 predictions
NBPredictions={}

# Iterate over the test document file by file
for i in range(0,len(test_data.filenames)):
    docs_test = [test_data.data[i]]
    # Apply the count vectorizer we used to fit the training data on the test data
    doc_test_counts = count_vect.transform(docs_test)
    # apply the tfidf transformation
    doc_test_tfidf = tfidf_transformer.transform(doc_test_counts)
    probability = clf.predict_proba(doc_test_tfidf).flatten()
    #print probability
    # sort according to the probability value in descending order
    a= (-probability).argsort()[:5]
    #create a list of predicted labels
    predicted=[]
    for target in a:
        predicted.append(training_data.target_names[target])

    true_label = test_data.target_names[test_data.target[i]]
    NBPredictions[i]=predicted
    if true_label in predicted:
        count+=1

print ' [Classification Ended: ' + str(datetime.datetime.now()) + ']'
print "ACCURACY : " + str(count/len(test_data.filenames))


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
From Occurences to Frequencies
Occurrence count is a good start but there is an issue: longer documents will have higher average count values than shorter documents, even though they might talk about the same topics.
To avoid these potential discrepancies it suffices to divide the number of occurrences of each word in a document by the total number of words in the document: these new features are called tf for Term Frequencies.
Another refinement on top of tf is to downscale weights for words that occur in many documents in the corpus and are therefore less informative than those that occur only in a smaller portion of the corpus.
This downscaling is called tf–idf for “Term Frequency times Inverse Document Frequency”.

 Refer to the following  Link  for a detailed explanation on TFIDF
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfTransformer
print '[process started: ' + str(datetime.datetime.now()) + ']'
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_count)
print '[process ended: ' + str(datetime.datetime.now()) + ']'
print X_train_tfidf.shape


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
Naive Bayes Classifier

I am not going to bore you with the details here. Let us just say that Naive Bayes is a very solid yet simple algorithm when it comes to classifying text and by default should be the very first algorithm that you must try out. Scikit has a very good implementation of Naive Bayes and we will be using it to classify our text. For the curious souls more about  NB Classifier .
Can you write Python code for it?

from sklearn.naive_bayes import MultinomialNB
print ' [Classification Started: ' + str(datetime.datetime.now()) + ']'
# you fit the NB model
clf = MultinomialNB().fit(X_train_tfidf, training_data.target)
print ' [Classification ended: ' + str(datetime.datetime.now()) + ']'


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
Notice how the number of features has gone up from 25834 in the previous case to 55469. This is because of using ngram features. We now use TFIDF on the word counts.
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfTransformer
print '[process started: ' + str(datetime.datetime.now()) + ']'
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_count)
print '[process ended: ' + str(datetime.datetime.now()) + ']'
print X_train_tfidf.shape

from sklearn.naive_bayes import MultinomialNB
print ' [Classification Started: ' + str(datetime.datetime.now()) + ']'
# you fit the NB model
clf = MultinomialNB().fit(X_train_tfidf, training_data.target)
print ' [Classification ended: ' + str(datetime.datetime.now()) + ']'

from __future__ import division

#load test data
test_data= load_files('/media/arvindramesh/CCF02769F02758CA/TextClassification-NewsGroup/test/',shuffle=True, encoding='ISO-8859-2')
# variable to track the accuracy
count=0
print ' [Classification Started: ' + str(datetime.datetime.now()) + ']'
# Iterate over the test document file by file
for i in range(0,len(test_data.filenames)):
    docs_test = [test_data.data[i]]
    # Apply the count vectorizer we used to fit the training data on the test data
    doc_test_counts = count_vect.transform(docs_test)
    # apply the tfidf transformation
    doc_test_tfidf = tfidf_transformer.transform(doc_test_counts)
    predicted = clf.predict(doc_test_tfidf)
    # Predicted Target label
    predicted_label=training_data.target_names[predicted]
    # True Target label of test document
    true_label=test_data.target_names[test_data.target[i]]
    # calculate the accuracy
    if predicted_label==true_label:
        count+=1
print ' [Classification Ended: ' + str(datetime.datetime.now()) + ']'
print "ACCURACY : " + str(count/len(test_data.filenames))


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
Now let us put it all together.
Gensim
Note that corpus that we loaded using load_files above resides fully in memory, as a plain Python Bunch. When the input corpus becomes very big with millions of documents , storing all of them in RAM may be a bad design decision and could crash the system. Gensim overcomes this by using a technique called streaming. While streaming a document is loaded and processed in memory one by one. Although the output is the same as for the plain Python bunch, the corpus is now much more memory friendly, because at most one vector resides in RAM at a time. The implementation of a gensim corpus class is as shown below:
Can you write Python code for it?

# to print the gensim logging in ipython notebook
import logging
logging.basicConfig(format='%(levelname)s : %(message)s', level=logging.INFO)
logging.root.level = logging.INFO  # ipython sometimes messes up the logging setup; restore

import gensim
import os
from os.path import join
from gensim import utils
import datetime, re, sys
from nltk.tokenize import RegexpTokenizer
from gensim import similarities,models, matutils
from gensim.similarities import MatrixSimilarity, SparseMatrixSimilarity, Similarity
from gensim import corpora
tokenizer = RegexpTokenizer(r'[a-zA-Z0-9]+')

# Tokenizer
def text_tokenize(text):
    tokens = tokenizer.tokenize(text.lower())
    # return the transformed corpus
    out=[]
    for token in tokens:
        # we filter out tokens that exist in stopwords list
        if token not in stopwords_list:
            out.append(token)

    return out  # returns a list

class MyCorpus(gensim.corpora.TextCorpus):
    def get_texts(self):
        for filename in self.input:
            yield text_tokenize(open(filename).read())


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
Now we can't begin to directly port the Reuters dataset. We need to segregate the training examples(documents) into its respective category folder for scikit to load it. Python OS module to the rescue!!
<img src="https://i.imgflip.com/p46f8.jpg" title="made at imgflip.com" width=300px, height=300px/>
Can you write Python code for it?

import os
import os.path
from os.path import join
import shutil
# root path of the dataset
rootpath = '/media/arvindramesh/CCF02769F02758CA/TextClassification-NewsGroup/reuters/'
# the cats.txt file that contains the mapping between the file and their respective categories
catpath= '/media/arvindramesh/CCF02769F02758CA/TextClassification-NewsGroup/reuters/cats.txt'
# path were the newly setup dataset will reside
newpath= '/media/arvindramesh/CCF02769F02758CA/TextClassification-NewsGroup/'
# using with to open a file will automatically handle the closing of file handler
with open(catpath) as catsfile:
    for line in catsfile:
        key = line.split()[0]  # path and filename
        value= line.split()[1] # category
        # create directory if it does not exists
        if not os.path.exists(join(newpath,key.split('/')[0],value)): os.makedirs(join(newpath,key.split('/')[0],value))
        #shutil.copy2(source,destination) lets you copy the files from the source directory to destination
        shutil.copy2(join(rootpath,key), join(newpath,key.split('/')[0],value))

print "DONE"


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
Testing the Performance of the Classifier
We now test the performance of the classifier in terms of the accuracy. What we do is we begin to import the test documents using the load_files the same way we did for the training documents. We then apply the CountVectorizer and TFIDF transformation to the test documents. The predict function will output the predicted label of the test document.
Can you write Python code for it?

from __future__ import division
import numpy as np
test_data= load_files('/media/arvindramesh/CCF02769F02758CA/TextClassification-NewsGroup/test/',shuffle=True, encoding='ISO-8859-2')
count=0
print ' [Classification Started: ' + str(datetime.datetime.now()) + ']'
for i in range(0,len(test_data.filenames)):
    docs_test = [test_data.data[i]]
    # Apply the count vectorizer we used to fit the training data on the test data
    doc_test_counts = count_vect.transform(docs_test)
    # apply the tfidf transformation
    doc_test_tfidf = tfidf_transformer.transform(doc_test_counts)
    # predict
    predicted = clf.predict(doc_test_tfidf)
    # Predicted label based on the classifier prediction above
    predicted_label=training_data.target_names[predicted]
    # True label of test document
    true_label=test_data.target_names[test_data.target[i]]
    # calculate the accuracy
    if predicted_label==true_label:
        count+=1
print ' [Classification Ended: ' + str(datetime.datetime.now()) + ']'
print "ACCURACY : " + str(count/len(test_data.filenames))


I am working on the file "A Noob's guide to text classification [Unrendered].ipynb".
The first task is:
The First Step
The first step in any machine learning problem is to go through the dataset and understand the structure. This will give us a better clarity when we start modelling the input data. We begin by extracting the dataset and notice that it contains a training, test folder. We observe that there are about 90 categories in the reuters dataset with about 7769 documents in the training set and 3019 documents in the test set. The distribution of categories in the corpus is highly skewed,
with 36.7% of the documents in the most common category, and only 0.0185% (2 documents) in each of the five least common categories. In fact, the original data source is even more skewed---in creating the corpus, any categories that did not contain at least one document in the training set and one document in the test set were removed from
the corpus by its original creator.
The Readme File should give you more information about the dataset.The cats.txt file contains the mapping of input filename to their respective category. There is also a stopword.txt file that contains a list of stop words. We will discuss more about the stopwords in the coming sections.

 The first thing to do is to load the dataset. Though there is CategorizedPlaintextCorpusReader function in nltk to do this, I am more inclined towards using the load_files function provided with scikit-learn to load the data. This is mainly because it is much easier to handle data in scikit as the load_files function returns a data bunch. Bunch in python let's you acess python dicts as objects.
Loading the dataset using Scikit-learn

Scikit-learn is an amazing library to quickly code up your machine learing project. It provides some very easy and useful functions to pretty much do everything from classification and regression to clustering. We are going to dwell right into scikit-learn and use the...
Can you write Python code for it?