I am working on the file "2016/python3/Day01.ipynb".
The first task is:
Tokenize the token into turn and number of blocks
The first letter represents the direction to turn to, and will only be 'L' for left, and 'R' for right.
The numbers after that are the number of blocks to walk.

Get mask based on direction to turn
The direction is the swapped direction, with the mask applied. So the previous (x,y) becomes (y,x) * mask(x,y).
The point is now the number of blocks added with the direction applied.
Part Two Check if we have visited the path. To do that, we must first identify whether we have moved along the X-axis or the Y-axis.
The movement_mask represents the mask applied to the last point to iterate over to the current point. This gives us every point along the path. The mask takes care of negatives and direction, and helps keep the logic clear.
If the x co-ordinates do not change, this is along the Y-axis.
Check whether we are moving towards the top or the bottom.
If the y attribute is increasing, we are moving towards the top, otherwise towards bottom.

If the y co-ordinates do not change, this is along the X-axis.
Check whether we are moving towards the right or the left.
If the x attribute is increasing, we are moving towards the right, otherwise towards left.

Now iterate through each point along the path
Check if there are any common co-ordinates on this path.
If the condition is satisfied, found a point lying in the intersecting path
Can you write Python code for it?

for token in data:
    turn, blocks = token[:1], int(token[1:])
    if turn == turns.left:
        mask = mask_left
    if turn == turns.right:
        mask = mask_right

    direction = VectorPoint(direction.y * mask.x, direction.x * mask.y)

    point = VectorPoint(
        point.x + blocks * direction.x,
        point.y + blocks * direction.y)

    if point.x == last_point.x:
        if point.y > last_point.y:
            movement_mask = VectorPoint(0, 1)
        else:
            movement_mask = VectorPoint(0, -1)
    else:
        if point.x > last_point.x:
            movement_mask = VectorPoint(1, 0)
        else:
            movement_mask = VectorPoint(-1, 0)

    last_point_holder = last_point

    while last_point.x != point.x or last_point.y != point.y:

        last_point = VectorPoint(
            last_point.x + movement_mask.x,
            last_point.y + movement_mask.y)

        for path in paths:

            if path.A.x == last_point.x and path.B.x == last_point.x:
                if abs(path.A.y - last_point.y)\
                        + abs(last_point.y - path.B.y)\
                        == abs(path.A.y - path.B.y):
                    break
            if path.A.y == last_point.y and path.B.y == last_point.y:
                if abs(path.A.x - last_point.x)\
                        + abs(last_point.x - path.B.x)\
                        == abs(path.A.x - path.B.x):
                    break
        else:
            # No paths match, move ahead.
            continue
        # Some path is found. Stop searching.
        break
    else:
        # Save the path.
        # `last_point = point` at this juncture
        path = Path(last_point_holder, point)
        last_point = point
        paths.append(path)
        continue
    # We found a path that has been visited
    distance = abs(last_point.x) + abs(last_point.y)
#     print(distance)
    break
else:
    print(0)


I am working on the file "2016/python3/Day01.ipynb".
The first task is:
When turning, we apply this mask (the logic if above).
Can you write Python code for it?

mask_left = VectorPoint(-1, 1)
mask_right = VectorPoint(1, -1)


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Answer lies in the variable code
Part Two
You finally arrive at the bathroom (it's a several minute walk from the lobby so visitors can behold the many fancy conference rooms and water coolers on this floor) and go to punch in the code. Much to your bladder's dismay, the keypad is not at all like you imagined it. Instead, you are confronted with the result of hundreds of man-hours of bathroom-keypad-design meetings:
    1
  2 3 4
5 6 7 8 9
  A B C
    D

You still start at "5" and stop when you're at an edge, but given the same instructions as above, the outcome is very different:
You start at "5" and don't move at all (up and left are both edges), ending at 5.
Continuing from "5", you move right twice and down three times (through "6", "7", "B", "D", "D"), ending at D.
Then, from "D", you move five more times (through "D", "B", "C", "C", "B"), ending at B.
Finally, after five more moves, you end at 3.
So, given the actual keypad layout, the code would be 5DB3.
Using the same instructions in your puzzle input, what is the correct bathroom code?
Solution logic
Since the keypad design has changed, we need to change our approach as well. In the new keypad, the grid is no longer symmetrical, so we cannot apply the mod logic to this keypad. The simplest approach would be to define each point in a dictionary with their co-ordinates, starting with 7 as the origin. This would allow the points to span in all four quadrants. Then we need to change the setters to accomodate the new co-ordinates, and finally, the numpad function to get the correct number using the dictionary.
Numbers
The numbers dictionary holds the co-ordinates for each key on the numpad. At the end, we can simply lookup the co-ordinate points and get the corresponding number. The number_keys allows the easier declaration of syntax, which is then inverted using the numbers_values dictionary.
Can you write Python code for it?


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Calculate Number
The updated method uses the new xy method to assign the values. Since we assume point to be global (in this scope), we directly use it.
Can you write Python code for it?

def calculate_number(line):
    for char in line:
        mask = directions[char]
        point.xy(point.x + mask.x, point.y + mask.y)


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Define a function that will parse the number from each line of input. We supply it the line.
Can you write Python code for it?

def calculate_number(line):
    for char in line:
        mask = directions[char]
        point.x += mask.x
        point.y += mask.y
    return numpad_number_from_point(point)

for line in data:
    code += calculate_number(line.strip())


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Directions
We must also update the directions, as we have changed the order of the numbers on the keypad. This involves reversing the sign of the UP and DOWN directions.
Can you write Python code for it?


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Directions
We use a dictionary to store the directions and the masks they represent. Each mask is a namedtuple having attributes x and y.
Can you write Python code for it?

from collections import namedtuple
Mask = namedtuple('Mask', ('x', 'y'))
directions = {
    'U': Mask(0, -1),
    'D': Mask(0, 1),
    'L': Mask(-1, 0),
    'R': Mask(1, 0),
}


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Keypad code
We run the keypad code over each line and store the result in keys.
Can you write Python code for it?

for line in data:
    calculate_number(line.strip())
    keys.append(numbers_values[Mask(point.x, point.y)])

answer = ''.join(keys)


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Point
Since each movement is along some direction, we go about this in a very pretentious and academic way. By declaring classes and objects and functions for everything. The end logic is a clean layout of the solution.
The Point class has attributes x and y. We use setter and getter to set the caps on these values.
Can you write Python code for it?

class Point:
    def __init__(self, x=0, y=0):
        self._x, self._y = x, y

    @property
    def x(self):
        return self._x

    @property
    def y(self):
        return self._y

    @x.setter
    def x(self, value):
        if 0 <= value <= 2:
            self._x = value

    @y.setter
    def y(self, value):
        if 0 <= value <= 2:
            self._y = value

    def __str__(self):
        return '(%s, %s)' % (self._x, self._y)


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Point
The starting point for this problem is 5, which we retrieve using the dictionary.
Can you write Python code for it?


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Point
We redefine the point class with updated setters for x and y. The new setters do nothing, and the actual update is done through the xy method, which takes in the two values. This checks whether the given values are in the correct range.
Can you write Python code for it?

class Point:
    def __init__(self, x=0, y=0):
        self._x, self._y = x, y

    @property
    def x(self):
        return self._x

    @property
    def y(self):
        return self._y

    @x.setter
    def x(self, value):
        pass

    @y.setter
    def y(self, value):
        pass

    def xy(self, x, y):
        if (
            (-2 <= x <= 2 and y == 0) or
            (-1 <= x <= 1 and -1 <= y <= 1) or
            (x == 0 and -2 <= y <= 2)):

            self._x, self._y = x, y

    def __str__(self):
        return '(%s, %s)' % (self._x, self._y)

    def __repr__(self):
        return '(%s, %s)' % (self._x, self._y)


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Reading in the input from file
Can you write Python code for it?

with open('../inputs/day02.txt', 'r') as f:
    data = f.readlines()


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
Solve
code holds the input code, and is a string.
Can you write Python code for it?


I am working on the file "2016/python3/Day02.ipynb".
The first task is:
keys holds the characters of the keypad code
Can you write Python code for it?


I am working on the file "2016/python3/Day03.ipynb".
The first task is:
Initialize the counter
Can you write Python code for it?


I am working on the file "2016/python3/Day03.ipynb".
The first task is:
Tokenize the input, then sort the tokens.
Can you write Python code for it?

tokenized_data = [
    [int(n) for n in line.split()]
    for line in data]
sorted_data = [sorted(line) for line in tokenized_data]

for a, b, c in sorted_data:
    if a + b > c:
        number_of_triangles += 1


I am working on the file "2016/python3/Day04.ipynb".
The first task is:
Apply the algorithm. The answer will be in sector_id_sum.
Can you write Python code for it?

from collections import Counter

sector_id_sum = 0

for line in data:
    match = pattern.match(line)
    characters, sector_id, checksum = match.groups()
    characters.replace('-','')
    character_counter = Counter(characters.replace('-', ''))
    character_checksum = ''.join(
        [x[0] for x in sorted(
                sorted(character_counter.items(), key=lambda x: x[0]),
                key=lambda x: x[1], reverse=True)])
    if character_checksum.startswith(checksum):
        sector_id_sum += int(sector_id)


I am working on the file "2016/python3/Day05.ipynb".
The first task is:
Part Two
As the door slides open, you are presented with a second door that uses a slightly more inspired security mechanism. Clearly unimpressed by the last version (in what movie is the password decrypted in order?!), the Easter Bunny engineers have worked out a better solution.
Instead of simply filling in the password from left to right, the hash now also indicates the position within the password to fill. You still look for hashes that begin with five zeroes; however, now, the sixth character represents the position (0-7), and the seventh character is the character to put in that position.
A hash result of 000001f means that f is the second character in the password. Use only the first result for each position, and ignore invalid positions.
For example, if the Door ID is abc:
- The first interesting hash is from `abc3231929`, which produces `0000015`...; so, `5` goes in position `1`: `_5______`.
- In the previous method, `5017308` produced an interesting hash; however, it is ignored, because it specifies an invalid position `(8)`.
- The second interesting hash is at index `5357525`, which produces `000004e`...; so, e goes in position `4`: `_5__e___`.

You almost choke on your popcorn as the final character falls into place, producing the password 05ace8e3.
Given the actual Door ID and this new method, what is the password? Be extra proud of your solution if it uses a cinematic "decrypting" animation.
Solution logic
Not much has changed from the last puzzle. We still calculate the MD5 hash as before. However, we now have an extra condition to check whether the hash is valid. The sixth character now represents the position of the character in the password. The seventh character is the actual password character. So we need some way to store password characters with positions. An easy approach would be a list pre-filled with eight None values. This makes it trivial to store the password characters by index.
password = [None for i in range(0, 8)]

To check whether the position parameter is valid, we need to...
Can you write Python code for it?

password_char_count = 0
password = [None for i in range(0, 8)]
hash_suffix = 0

while password_char_count != 8:
    string = door_id + str(hash_suffix)
    hash_suffix += 1
    md5 = hashlib.md5()
    md5.update(string.encode('ascii'))
    md5hash = md5.hexdigest()
    if not md5hash.startswith('00000'):
        continue
    if not md5hash[5].isdigit():
        continue
    position_char = int(md5hash[5])
    if 0 <= position_char <= 7 and password[position_char] is None:
        password[position_char] = md5hash[6]
        password_char_count += 1
''.join(password)


I am working on the file "2016/python3/Day06.ipynb".
The first task is:
Get the most common element from each counter. Since the most_common method returns a list of tuples (key-value), we need to access the first element from each of them respectively. And finally join them together into a string to get the string.
Can you write Python code for it?

answer = ''.join([counter.most_common(1)[0][0] for counter in counters])


I am working on the file "2016/python3/Day06.ipynb".
The first task is:
Part Two
Of course, that would be the message - if you hadn't agreed to use a modified repetition code instead.
In this modified code, the sender instead transmits what looks like random data, but for each character, the character they actually want to send is slightly less likely than the others. Even after signal-jamming noise, you can look at the letter distributions in each column and choose the least common letter to reconstruct the original message.
In the above example, the least common character in the first column is a; in the second, d, and so on. Repeating this process for the remaining characters produces the original message, advent.
Given the recording in your puzzle input and this new decoding methodology, what is the original message that Santa is trying to send?
Solution logic
Small change - instead of most common, we now need the least common characters from the counters. Thankfully, since the counter returns a nice sorted list in most_common, we can use that to get the least common element - it's the one that is last!!!
characters = counter.most_common()[-1]

That being done, we combine the characters back into a string to get the answer.
Can you write Python code for it?

answer = ''.join([counter.most_common()[-1][0] for counter in counters])


I am working on the file "2016/python3/Day06.ipynb".
The first task is:
Reading characters from input, and counting them. To add an item to the counter, simply increment its value.
Can you write Python code for it?

for line in data:
    for index, char in enumerate(line):
        counters[index][char] += 1


I am working on the file "2016/python3/Day07.ipynb".
The first task is:
A little testing using the test data from the problem.
Can you write Python code for it?

test_data = [
    'abba[mnop]qrst',
    'abcd[bddb]xyyx',
    'aaaa[qwer]tyui',
    'ioxxoj[asdfgh]zxcvbn',
]
for line in test_data:
    tokens = tokenize(line)
    bracketed = []
    non_bracketed = []
    for token in tokens:
        if token.startswith('[') and token.endswith(']'):
            bracketed.append(token)
        else:
            non_bracketed.append(token)
    if any(check_abba(token) for token in bracketed):
        continue
    if any(check_abba(token) for token in non_bracketed):
        print(line, True)


I am working on the file "2016/python3/Day07.ipynb".
The first task is:
Part Two
You would also like to know which IPs support SSL (super-secret listening).
An IP supports SSL if it has an Area-Broadcast Accessor, or ABA, anywhere in the supernet sequences (outside any square bracketed sections), and a corresponding Byte Allocation Block, or BAB, anywhere in the hypernet sequences. An ABA is any three-character sequence which consists of the same character twice with a different character between them, such as xyx or aba. A corresponding BAB is the same characters but in reversed positions: yxy and bab, respectively.
For example:
- `aba[bab]xyz` supports SSL (`aba` outside square brackets with corresponding `bab` within square brackets).
- `xyx[xyx]xyx- does not support SSL (`xyx`, but no corresponding `yxy`).
- `aaa[kek]eke` supports SSL (`eke` in supernet with corresponding `kek` in hypernet; the `aaa` sequence is not related, because the interior character must be different).
- `zazbz[bzb]cdb` supports SSL (`zaz` has no corresponding `aza`, but `zbz` has a corresponding `bzb`, even though `zaz` and `zbz` overlap).

How many IPs in your puzzle input support SSL?
Solution logic
Instead of having a four-character palindrome, we now have two three-character palindromes, where their characters are interchanged. For any three-letter palindrome in a non-bracketed string, we have to check whether a corresponding palindrome exists in any bracketed string. If it exists, the IP supports SSL.
Algorithm:
main:
    - set counter = 0
    - for each line of input:
        - tokenize line in to bracketed and non-bracketed tokens
        - for each token in non-bracketed tokens:
            - get ABA sequence
            - for each token in bracketed tokens:
                - check for BAB sequence
                - if present, increment counter, move to next line
check_aba:
    - accepts string input
    - returns a generator for every ABA sequence
    - for i in range 0 to length of string - 2:
        - if first character is same as third character but is different than second character:
   ...
Can you write Python code for it?

def check_aba(string):
    if len(string) < 3:
        return []
    for i in range(0, len(string) - 2):
        if string[i] == string[i + 2] and string[i] != string[i + 1]:
            yield string[i] + string[i + 1] + string[i + 2]

def check_bab(string, token_list):
    string = string[1] + string[0] + string[1]
    return any(string in token for token in token_list)


I am working on the file "2016/python3/Day07.ipynb".
The first task is:
Since the above works with test data, we go ahead and load the problem inputs.
Can you write Python code for it?

with open('../inputs/day07.txt', 'r') as f:
    data = [line.strip() for line in f.readlines()]

counter = 0
for line in data:
    tokens = tokenize(line)
    bracketed = []
    non_bracketed = []
    for token in tokens:
        if token.startswith('[') and token.endswith(']'):
            bracketed.append(token)
        else:
            non_bracketed.append(token)
    if any(check_abba(token) for token in bracketed):
        continue
    if any(check_abba(token) for token in non_bracketed):
        counter += 1


I am working on the file "2016/python3/Day08.ipynb".
The first task is:
Counting number of pixels that are ON
Can you write Python code for it?

def display_pixels_lit(display):
    '''return number of pixels that are ON'''
    return sum(sum(row) for row in display)

display_pixels_lit(display)


I am working on the file "2016/python3/Day08.ipynb".
The first task is:
Part Two
You notice that the screen is only capable of displaying capital letters; in the font it uses, each letter is 5 pixels wide and 6 tall.
After you swipe your card, what code is the screen trying to display?
Solution logic
To get the letters without any fuss, we simply print out the display on screen and then identify what letters are present and type them out. No fancy letter shapes stored in data structures matched at runtime.
Can you write Python code for it?

def print_display(display):
    for row in display:
        print(''.join(['#' if i else '.' for i in row]))

print_display(display)


I am working on the file "2016/python3/Day08.ipynb".
The first task is:
Retrieving the input
Can you write Python code for it?

with open('../inputs/day08.txt', 'r') as f:
    data = [line.strip() for line in f.readlines()]
#     TEST DATA from problem description
#     data = [
#         'rect 3x2',
#         'rotate column x=1 by 1',
#         'rotate row y=0 by 4',
#         'rotate column x=1 by 1'
#     ]

for line in data:
    for pattern, handler in instructions.items():
        match = pattern.match(line)
        if match:
            inputs = [int(i) for i in match.groups()]
            handler(display, *inputs)
            break
    else:
        print('ERROR', line)


I am working on the file "2016/python3/Day09.ipynb".
The first task is:
Reading in markers, Calculating decompressed length: We use the (very awesome) itertools module to do the iterating and filtering for us.
We use an iterator to go over the input values, so that we can use the itertools functions such as takewhile that selects characters as long as a condition is fulfilled.
def takewhile(condition, data):
    filtered_data = []
    for item in data:
        if condition(data):
            filtered_data.append(item)
        else:
            break
    return filtered_data

We use takewhile to swallow characters until we reach the markers, and then to get the marker itself. Using regular expressions, we extract the two values from the marker. Since we are using iterators, we need to skip the next A characters, which we do using a for loop.
At the end, the answer is in the count variable.
Can you write Python code for it?

from itertools import islice, takewhile
import re
numbers = re.compile(r'(\d+)')

def decompress(data_iterator):
    '''parses markers and returns index of last character and length of decompressed data'''
    count = 0
    index = 0

    while True:
        #  handle single tokens that decompress to length 1 until start of marker
        count += len(list(takewhile(lambda character: character != '(', data_iterator)))
        #  extract marker
        marker = ''.join(takewhile(lambda character: character != ')', data_iterator))
        #  extract A and B
        try:
            a, b = map(int, numbers.findall(marker))
        except ValueError:
            # EOF or no other markers present
            break
        # skip the next a characters
        for i in range(a):
            next(data_iterator)
        # increment count
        count += a * b

    return count

print(decompress(iter(data)))


I am working on the file "2016/python3/Day10.ipynb".
The first task is:
Actions for bots - giving out chips
Each bot will only act when it has two chips, and it will give those two chips according to the rules specified. The function returns True if the specified action was performed, False otherwise.
Can you write Python code for it?

def bot_chip_handler(bot_number):
    # retrieve the bot
    bot = bot_list[bot_number]

    # check if it has two chips
    if len(bot.chips) < 2:
        return False

    # check if it has the specified chip
    if 61 in bot.chips and 17 in bot.chips:
        print('answer', bot.number)

    # check if it has no instructions, which means no actions
    if bot.give_low is None and bot.give_high is None:
        return False

    # give low chip
    if bot.give_low.type == 'bot':
        # give chip to bot
        bot_list[bot.give_low.number].chips.append(bot.chips[0])
        bot_list[bot.give_low.number].chips.sort()
    else:
        # give chip to output bin
        output_bins[bot.give_low.number].append(bot.chips[0])

    # give high chip
    if bot.give_high.type == 'bot':
        # give chip to bot
        bot_list[bot.give_high.number].chips.append(bot.chips[1])
        bot_list[bot.give_high.number].chips.sort()
    else:
        # give chip to output bin
        output_bins[bot.give_high.number].append(bot.chips[1])

    # remove chips from bot
    bot.chips = []
    # return action succesfull
    return True


I am working on the file "2016/python3/Day10.ipynb".
The first task is:
Class Bot to hold the bot instances
Can you write Python code for it?

class Bot:
    def __init__(self, number):
        self.number = number
        self.chips = []
        self.give_low = None
        self.give_high = None

    def __repr__(self):
        return 'number: %s chips: %s low: %s high: %s' % (self.number, self.chips, self.give_low, self.give_high)


I am working on the file "2016/python3/Day10.ipynb".
The first task is:
Creating the necessary data structures and regex patterns.
The bot instructions each contain a word specifying who the chip is from/going to and the chip number
We use this information to create the instruction_type structure to hold the type and number of the instruction.
Can you write Python code for it?

from collections import namedtuple
instruction_type = namedtuple('instruction_type', ('type', 'number'))

import re

bot_instruction = re.compile('(\w+) (\d+)')
input_instruction = re.compile('(\d+)')

bot_list = {}
output_bins = {}


I am working on the file "2016/python3/Day10.ipynb".
The first task is:
Helper functions
The ensure... functions are there to help (or clear up the code) when accessing bots and output bins as they are encountered in the instructions. Another (elegant?) way to do this would be to wrap each of the calls to the dictionary in a try...except and catch IndexError for invalid accesses.
Can you write Python code for it?

def ensure_bot_is_registered(bot_number):
    if bot_number not in bot_list:
        bot_list[bot_number] = Bot(bot_number)

def ensure_bin_is_registered(bin_number):
    if bin_number not in output_bins:
        output_bins[bin_number] = []


I am working on the file "2016/python3/Day10.ipynb".
The first task is:
Part Two
What do you get if you multiply together the values of one chip in each of outputs 0, 1, and 2?
Solution logic
We need to multiple one of the values in output bins 0, 1, and 2. This means either that they contain only one value or that we have to choose only the first one. In any case, we access only the first element in each bin.
reduce(mul, [output_bins[i][0] for i in range(3)])

This is the functional way to write -
n = 1
for i in range 0 to 2 (3 - 1 or i < 3)
    n *= output[i][0]
Can you write Python code for it?

from functools import reduce
from operator import mul
print('answer', reduce(mul, [output_bins[i][0] for i in range(3)]))


I am working on the file "2016/python3/Day10.ipynb".
The first task is:
main
Here we take each in the given instructions, identify whether the line is an instruction for the bot, or an action where bots are given/take chips from the input bin. The instruction lines start with bot and the action lines start with value.
For bot instructions, the appropriate bot is retrieved from list, its rules are added to it.
For actions, the bot is retrieved and the chip is added to it. The chip handler function is then run to determine what to do with the bot.
At the end of each iteration, we go from each both in increasing order by running the chip handler, and if any bot gives out chip to another bot, we start again, since the bot can give chips to other bots with a number lesser than theirs.
Can you write Python code for it?

for line in data:
    if line.startswith('bot'):
        # this is an instruction for the bot
        # get the tokens from the line
        main_bot, give_low, give_high = [
            # the first is a word, the second is the bot/bin number
            instruction_type(token[0].strip(), int(token[1]))
            for token in bot_instruction.findall(line)]

        # retrieve the bot
        ensure_bot_is_registered(main_bot.number)
        main_bot = bot_list[main_bot.number]

        # handle low/high chip rules
        # and whether they are for bots or bins
        if give_low.type == 'bot':
            ensure_bot_is_registered(give_low.number)
        else:
            ensure_bin_is_registered(give_low.number)
        main_bot.give_low = give_low
        if give_high.type == 'bot':
            ensure_bot_is_registered(give_high.number)
        else:
            ensure_bin_is_registered(give_high.number)
        main_bot.give_high = give_high

    else:
        # this is an action where bots are given chips from the input bin
        # there are only two values - the chip value and the bot number
        chip_number, bot_number = map(int, input_instruction.findall(line))
        ensure_bot_is_registered(bot_number)
        bot = bot_list[bot_number]
        bot.chips.append(chip_number)
        bot.chips.sort()
        # run handler on this bot since it could now have two chips
        bot_chip_handler(bot.number)

    # run handlers on all bots in an increasing order
    # this is an arbitary choice, it is not specified in the input
    condition = True
    while condition:
        condition = False
        for bot_number in sorted(bot_list.keys()):
            returned_flag = bot_chip_handler(bot_number)
            if returned_flag == True:
                condition = True
                break


I am working on the file "2016/python3/Day13.ipynb".
The first task is:
Algorithm
Doing everything as before, except for a few changes.

pop the step off the queue or get the first element
if it has reached 50 steps, do not process it more. Move on the to the next move.
calculate moves for this point in each direction
add the direction to the current point
if the point is not valid, skip it
calculate cost of next move as 1 + cost of current step
if the point has not been visited before or the current cost is lower than that in the index
append the move to the queue
update the cost for current point in the index
add the point to the visited set

Here, we prioritize the paths that have shorter steps to reach the same point, since shorter steps mean more points traversed (overall). Therefore, we only add the point to the set if we are on optimum paths. This allows us to spread as much as possible without overlapping points or moves.
Can you write Python code for it?

queue = []
points_traversed = {}
queue.append(initial_state)
points_traversed[start_point] = 0

while queue:
    state = queue.pop(0)
    if state.steps == 50:
        continue
    for vector in directions:
        direction = Point(*vector)
        move = Point(
            state.point.x + direction.x,
            state.point.y + direction.y)
        if not valid_point(move):
            continue
        cost = state.steps + 1
        if move not in points_traversed or cost < points_traversed[move]:
            queue.append(Move(state.steps + 1, move))
            points_traversed[move] = cost
            visited.add(move)

print('answer', len(visited))


I am working on the file "2016/python3/Day13.ipynb".
The first task is:
Branching
We need to store the paths and whether we have been there before, and if yes, then which route were we taking. A simple queue should suffice.
The approach we are going to use is that of breadth-first-search. In this approach, we identify all possible directions we can currently move in and put them in the queue. Then while the stack is not empty, we pop a move out, and repeat the directions it can take and so on until we reach the end-point, at which, we store the length of the path we took.
To store the length of the path, each move put on the stack must have an accompanying count of the steps it took to reach there. It is possible to apply heuristic by storing an index of paths sorted by the length and giving priority to pursuing paths with a shorter length first, or those that are nearer to the end-point.
One very important thing to note: we must not travel to the same point again - otherwise we'll keep chasing our own tails in recursive cycles (or an infinite while loop, take your pick). To prevent that, we have to maintain an index of points we have already traversed. We use a dictionary to hold this index, which stores the number of steps we took to traverse to this point. If there comes a move that reaches the same point in lesser number of steps, we know that it is a shorter route, and we continue traversing it. If the move has more number of steps to reach the same point, then it is not an optimum path, and do not need to continue along it.
Can you write Python code for it?


I am working on the file "2016/python3/Day13.ipynb".
The first task is:
Building the office structure
We store the office points in a namedtuple called Point with the properties x and y to hold the co-ordinates.
Can you write Python code for it?

from collections import namedtuple
Point = namedtuple('Point', ('x', 'y'))


I am working on the file "2016/python3/Day13.ipynb".
The first task is:
Directions
At any given moment, there are four possible directions - up, down, left, and right. As per the co-ordinate system, we can map them as:
d      X    Y
-------------
up     0,  +1
down   0,  -1
left  -1,   0
right +1,   0

Thus the number of possible movements become any permutation of (-1, 0, 1) of length 2. We can get this via:
permutations of (-1, 0, 1) with length 2 if sum != 0

Out of these, we need to remove that might result in an out-of-bounds position. Given (xc, yc) as the current position, and (xd, yd) as the direction vector:
direction is valid if 0 <= xc + xd and 0 <= yc + yd

We do not need to check for an upper bound as the space extends infinitely.
Can you write Python code for it?

from itertools import permutations
directions = tuple(
    x for x in permutations((-1, 0, 1), 2)
    if sum(x) != 0)
print(directions)


I am working on the file "2016/python3/Day13.ipynb".
The first task is:
Part Two
How many locations (distinct x,y coordinates, including your starting location) can you reach in at most 50 steps?
Solution logic
At first glance, it was not clear to me whether I need to calculate the most distinct points that could be taken in any route in 50 steps, or ALL distinct points from all paths possible in 50 steps. A little clear headed thought said that the most steps possible in 50 moves is 50 + 1 = 51 including the starting step. So that could not be the answer. So it was the other option - to calculate all distinct points visited by all paths possible in 50 moves.
Storing distinct points
Whenever the word distinct occurs, always think of sets. The set starts with the start point as we are currently there, and have visited it.
Can you write Python code for it?

visited = set()
visited.add(start_point)


I am working on the file "2016/python3/Day13.ipynb".
The first task is:
Run
Running the algorithm as long as there are moves in the queue or we have not reached our end point, we follow the steps as:

pop the step off the queue or get the first element
if it has reached the end poin, we have received our solution. STOP.
calculate moves for this point in each direction
add the direction to the current point
if the point is not valid, skip it
calculate cost of next move as 1 + cost of current step
if the point has not been visited before or the current cost is lower than that in the index
append the move to the queue
update the cost for current point in the index
Can you write Python code for it?

while queue:
    state = queue.pop(0)
    if state.point == end_point:
        break
    for vector in directions:
        direction = Point(*vector)
        move = Point(
            state.point.x + direction.x,
            state.point.y + direction.y)
        if not valid_point(move):
            continue
        cost = state.steps + 1
        if move not in points_traversed or cost < points_traversed[move]:
            queue.append(Move(state.steps + 1, move))
            points_traversed[move] = cost

print('answer', state.steps)


I am working on the file "2016/python3/Day14.ipynb".
The first task is:
Part Two
Of course, in order to make this process even more secure, you've also implemented key stretching.
Key stretching forces attackers to spend more time generating hashes. Unfortunately, it forces everyone else to spend more time, too.
To implement key stretching, whenever you generate a hash, before you use it, you first find the MD5 hash of that hash, then the MD5 hash of that hash, and so on, a total of 2016 additional hashings. Always use lowercase hexadecimal representations of hashes.
For example, to find the stretched hash for index 0 and salt abc:

Find the MD5 hash of abc0: 577571be4de9dcce85a041ba0410f29f.
Then, find the MD5 hash of that hash: eec80a0c92dc8a0777c619d9bb51e910.
Then, find the MD5 hash of that hash: 16062ce768787384c81fe17a7a60c7e3.
...repeat many times...
Then, find the MD5 hash of that hash: a107ff634856bb300138cac6568c0f24.

So, the stretched hash for index 0 in this situation is a107ff.... In the end, you find the original hash (one use of MD5), then find the hash-of-the-previous-hash 2016 times, for a total of 2017 uses of MD5.
The rest of the process remains the same, but now the keys are entirely different. Again for salt abc:

The first triple (222, at index 5) has no matching 22222 in the next thousand hashes.
The second triple (eee, at index 10) hash a matching eeeee at index 89, and so it is the first key.
Eventually, index 22551 produces the 64th key (triple fff with matching fffff at index 22859.

Given the actual salt in your puzzle input and using 2016 extra MD5 calls of key stretching, what index now produces your 64th one-time pad key?
Solution logic
We only need to change the definition of get_hash_string to calculate the hash 2016 times more. And then simply run the algorithm again.
To prevent computationally intensive operations from repeating themselves, we maintain an index of hashes so that we can easily lookup indexes without needing to calculate their hashes.
Can you write Python code for it?

hash_index = {}

def get_hash_string(key):
    if key in hash_index:
        return hash_index[key]
    string = '{salt}{key}'.format(salt=salt, key=key)
    md5 = hashlib.md5()
    md5.update(string.encode('ascii'))
    hashstring = md5.hexdigest()
    # PART TWO
    for i in range(0, 2016):
        md5 = hashlib.md5()
        md5.update(hashstring.encode('ascii'))
        hashstring = md5.hexdigest()
    hash_index[key] = hashstring
    return hashstring

print('answer', run()[63])


I am working on the file "2016/python3/Day15.ipynb".
The first task is:
Part Two
After getting the first capsule (it contained a star! what great fortune!), the machine detects your success and begins to rearrange itself.
When it's done, the discs are back in their original configuration as if it were time=0 again, but a new disc with 11 positions and starting at position 0 has appeared exactly one second below the previously-bottom disc.
With this new disc, and counting again starting from time=0 with the configuration in your puzzle input, what is the first time you can press the button to get another capsule?
Solution logic
Since there is no change in logic, we simply add the new disc to the disc list and run the algorithm again.
Can you write Python code for it?

discs = input_data
discs.append(Disc(11, 0))

print('asnwer', run())


I am working on the file "2016/python3/Day15.ipynb".
The first task is:
Reading the input and creating discs
Can you write Python code for it?

from copy import deepcopy
import re
numbers = re.compile('(\d+)')

with open('../inputs/day15.txt', 'r') as f:
    for line in f.readlines():
        disc_no, total_positions, time, starting_position =\
            map(int, numbers.findall(line))
        discs.append(Disc(total_positions, starting_position))
    # preserve input data
    input_data = deepcopy(discs)

for index, disc in enumerate(discs):
    print('disc#', index, disc)


I am working on the file "2016/python3/Day15.ipynb".
The first task is:
We declare a list to hold the discs according to the specified order.
Can you write Python code for it?


I am working on the file "2016/python3/Day15.ipynb".
The first task is:
We now iterate through each time unit to find when the capsule can be dropped through all discs. Ideally, there should not be an infinite loop, but if required, just set it to some VERY high index so that it will terminate at some point.
Can you write Python code for it?

def run():
    for time in range(0, 10000000):
        if not any((
                (discs[i].current_position + 1 + i) % discs[i].total_positions
                for i in range(0, len(discs)))):
            return time
            break
        for disc in discs:
            disc.current_position = (disc.current_position + 1) % disc.total_positions

print('answer', run())


I am working on the file "2016/python3/Day16.ipynb".
The first task is:
Part Two
The second disk you have to fill has length 35651584. Again using the initial state in your puzzle input, what is the correct checksum for this disk?
Can you write Python code for it?

DISK_LENGTH = 35651584

string = fill_disk(input_data)

print('string length: ', len(string))

checksum = create_checksum(string)

print('checksum length', len(checksum))
print(checksum)


I am working on the file "2016/python3/Day17.ipynb".
The first task is:
Checking for open doors
We take the first four characters of the checksum, compare them against the given condition of being less than 10 or 'a' which signlas a closed door. Return only directions that are valid - open doors.
'a' or less --> closed
others      --> open
Can you write Python code for it?

def open_doors(checksum):
    def is_open(character):
        if character > 'a':
            return True
        return False

    return [
        directions[index]
        for index, character in enumerate(checksum[:4])
        if is_open(character)]


I am working on the file "2016/python3/Day17.ipynb".
The first task is:
Checksum
Calculating the checksum is simply taking the MD5 hash of it.
Can you write Python code for it?

import hashlib

def md5(string):
    md5 = hashlib.md5()
    md5.update(string.encode('ascii'))
    return md5.hexdigest()


I am working on the file "2016/python3/Day17.ipynb".
The first task is:
Directions
We maintain a tuple of points for directions. This allows the direction to match the specific character in the checksum.
1st character signals open door --> 1st element in tuple gives direction
Can you write Python code for it?


I am working on the file "2016/python3/Day17.ipynb".
The first task is:
Heap
We declare a states list for holding the heap. The heap contains only the starting point at this time. While the length of the path string at this point is the length of the input, it is not a valid answer nor reflects the final path string. I chose to use the length of the entire string so as to avoid calculating the difference between the input string and the direction string everytime. It also makes code easier to read, IMHO.
Can you write Python code for it?

from heapq import heappush, heappop

states = []
heappush(states, (len(start_point.path), start_point))


I am working on the file "2016/python3/Day17.ipynb".
The first task is:
Point
As with before, having a namedtuple makes the code cleaner and easier to read.
Can you write Python code for it?

from collections import namedtuple
Point = namedtuple('Point', ('x', 'y', 'path'))


I am working on the file "2016/python3/Day17.ipynb".
The first task is:
Start and End states
It is very important to know what exactly is the starting and ending state of the algorithm. In this case, the ending state is the room at the very bottom right, with the co-ordinates (3, 3). Since we do not know its path, and it is not required to declare it, we use None instead.
Can you write Python code for it?

start_point = Point(0, 0, path_string)
end_point = Point(3, 3, None)


I am working on the file "2016/python3/Day17.ipynb".
The first task is:
Validating points / moves
A point is valid if it stays within the boundaries of the 4 x 4 matrix.
Can you write Python code for it?

def valid_point(point):
    if 0 <= point.x < 4 and 0 <= point.y < 4:
        return True


I am working on the file "2016/python3/Day18.ipynb".
The first task is:
Edge cases
There are edge cases when generating tiles that are in the first or the last position. In the first position, the left upper tile does not exist, in which case, we assume it to be safe. Same for the upper right tile for the tile in the last position.
We create an accessor method to get us the correct tile even when the index is out of bounds. Here we assume that all tiles out of bounds are 'safe'.
Can you write Python code for it?

def get_tile(row, index):
    if 0 <= index < len(row):
        return row[index]
    return SAFE


I am working on the file "2016/python3/Day18.ipynb".
The first task is:
Generating the 10 x 10 test data
Can you write Python code for it?

row = '.^^.^.^^^^'
print(row)
for _ in range(10 - 1):
    row = ''.join((make_tile(row, index) for index in range(len(row))))
    print(row)


I am working on the file "2016/python3/Day18.ipynb".
The first task is:
Part Two
How many safe tiles are there in a total of 400000 rows?
Can you write Python code for it?

safe_tiles = input_data.count(SAFE)
row = input_data
for _ in range(400000 - 1):
    row = ''.join((make_tile(row, index) for index in range(len(row))))
    safe_tiles += row.count(SAFE)
print('answer', safe_tiles)


I am working on the file "2016/python3/Day18.ipynb".
The first task is:
Running against the given input for 40 rows
Can you write Python code for it?

with open('../inputs/Day18.txt', 'r') as f:
    input_data = f.readline().strip()

safe_tiles = input_data.count(SAFE)
row = input_data
for _ in range(40 - 1):
    row = ''.join((make_tile(row, index) for index in range(len(row))))
    safe_tiles += row.count(SAFE)
print('answer', safe_tiles)


I am working on the file "2016/python3/Day18.ipynb".
The first task is:
Test data
Can you write Python code for it?

row = '..^^.'
print(row)
next_row = ''.join((make_tile(row, index) for index in range(len(row))))
print(next_row)
next_row = ''.join((make_tile(next_row, index) for index in range(len(next_row))))
print(next_row)


I am working on the file "2016/python3/Day19.ipynb".
The first task is:
Alternative logic
We use the concept of balanced trees, where we split the elves into two different lists. In one list, we add the first half elements, and in the second list, we add the other half in the reverse order, i.e. append to the front. Since the elves are always removed from the middle of the list, we can do this easily in a tree, as the middle is always the root. After removal, we need to balance the tree by moving elements from the top of the left branch (the starting elves) to the top of the right branch (the last elves), and then move the elements from the bottom of the right branch (the middle elves) to the bottom of the left branch (the middle elves).
left branch  --> first half of elements, normal order
right branch --> second half of elements, reverse order

while length of tree (left + right) > 1:
    if left tree has more nodes:
        remove node from end of left branch
    else:
        remove node from end of right branch
    remove node from top of left branch and append to top of right branch
    remove node from bottom of right branch and append to bottom of left branch

In python, lists are great data structures, but they are not performant when it comes to removing things from the front. Since we need to remove things from both ends, we will use dequeue which is a double ended queue and is performant for insertions and removals at both ends, but will perform badly for other operations such as splicing and random removals (which we do not require).
Can you write Python code for it?

from collections import deque
left = deque(i for i in range(1, input_data//2 + 1))
right = deque(i for i in range(input_data, input_data//2, -1))
print('tree left->', len(left), '1:', left[0], 'last:', left[-1])
print('tree right->', len(right), '1:', right[0], 'last:', right[-1])

while left and right:
    if len(left) > len(right):
        left.pop()
    else:
        right.pop()
    right.appendleft(left.popleft())
    left.append(right.pop())


I am working on the file "2016/python3/Day19.ipynb".
The first task is:
Part Two
Realizing the folly of their present-exchange rules, the Elves agree to instead steal presents from the Elf directly across the circle. If two Elves are across the circle, the one on the left (from the perspective of the stealer) is stolen from. The other rules remain unchanged: Elves with no presents are removed from the circle entirely, and the other elves move in slightly to keep the circle evenly spaced.
For example, with five Elves (again numbered 1 to 5):
The Elves sit in a circle; Elf 1 goes first:
      1
    5   2
     4 3
Elves 3 and 4 are across the circle; Elf 3's present is stolen, being the one to the left. Elf 3 leaves the circle, and the rest of the Elves move in:
  1           1
5   2  -->  5   2
 4 -          4

Elf 2 steals from the Elf directly across the circle, Elf 5:
  1         1
-   2  -->     2
  4         4

Next is Elf 4 who, choosing between Elves 1 and 2, steals from Elf 1:
 -          2
    2  -->
 4          4

Finally, Elf 2 steals from Elf 4:
 2
    -->  2
 -

So, with five Elves, the Elf that sits starting in position 2 gets all the presents.
With the number of Elves given in your puzzle input, which Elf now gets all the presents?
Solution logic
We need to redefine our play_round function, where instead of only skipping over the next elf, we need to remove the middle elf from the list. Which one is the middle elf depends on the length of the list at that time. So at each turn (and not round), the length of the list changes. Since we are implmenting the naive solution, we will iterate over the list once in the loop instead of optimizing the index access.
Since we need to eliminate the middle elf at each turn, we must find the middle elf. Let us start by seeing what the pattern is by plotting the number of elfs with the middle elf (the one to be eliminated) for each arrangement.
n    0-indexed array
1         -
2         1
3         1
4         2
5         2
6         3
7         3
8         4
9         4

We can see...
Can you write Python code for it?

# DO NOT RUN
def do_not_run():
    elves = [elf for elf in range(1, input_data + 1)]
    while len(elves) > 1:
        elves.pop(len(elves)//2)
        elves = elves[1:] + elves[:1]
    print(elves[0])


I am working on the file "2016/python3/Day19.ipynb".
The first task is:
Simulate one round of stealing presents
To simulate one round, we define the function play_round that takes an elf, then skips one over while iterating over the list. If the number of elves are odd, then the first elf's presents are stolen by the last elf, and therefore, we drop the first elf from the list.
Can you write Python code for it?

def play_round(elves):
    _elves = []
    elf = 0
    while elf < len(elves):
        _elves.append(elves[elf])
        elf += 2
    if len(elves) % 2 == 1:
        _elves.pop(0)
    return _elves


I am working on the file "2016/python3/Day19.ipynb".
The first task is:
Since we do not know which of the left or right branches might contain the final element, we need to check which of the queues are not empty and then retrieve the answer from that.
Can you write Python code for it?

if left:
    print('answer', left[0])
else:
    print('answer', right[0])


I am working on the file "2016/python3/Day20.ipynb".
The first task is:
Part Two
How many IPs are allowed by the blacklist?
Solution logic
Instead of breaking if none of the rules match, we keep a counter that we increment to count the number of IPs.
Can you write Python code for it?

rule_index = 0
n = 0
ips = 0

while n <= UPPER_LIMIT:
    for i in range(rule_index, len(data)):
        low, high = data[i]
        if low <= n <= high:
            n = high + 1
            break
    else:
        ips += 1
        n += 1

print(ips)


I am working on the file "2016/python3/Day21.ipynb".
The first task is:
Defining rules and actions
Can you write Python code for it?

import re
rules = {
    'swap-position': re.compile(r'^swap position (\d+) with position (\d+)'),
    'swap-letter': re.compile(r'^swap letter (\w+) with letter (\w+)'),
    'rotate-letter': re.compile(r'^rotate based on position of letter (\w)+'),
    'rorate-right': re.compile(r'^rotate right (\d+) step'),
    'rotate-left': re.compile(r'^rotate left (\d+) step'),
    'move': re.compile(r'^move position (\d+) to position (\d+)'),
    'reverse': re.compile(r'^reverse positions (\d+) through (\d+)')
}
actions = {
    'swap-position': swap_position,
    'swap-letter': swap_letter,
    'rotate-letter': rotate_letter,
    'rorate-right': rotate_right,
    'rotate-left': rotate_left,
    'move': move,
    'reverse': reverse
}


I am working on the file "2016/python3/Day21.ipynb".
The first task is:
Getting input
Can you write Python code for it?

with open('../inputs/day21.txt', 'r') as f:
        input_data = f.readlines()


I am working on the file "2016/python3/Day21.ipynb".
The first task is:
Part Two
You scrambled the password correctly, but you discover that you can't actually modify the password file on the system. You'll need to un-scramble one of the existing passwords by reversing the scrambling process.
What is the un-scrambled version of the scrambled password fbgdceah?
Solution logic
Rather than going through the effort of writing the reverse operations for every instruction, we simply iterate through all the permutations of the letters a, b, c, d, e, g, h and see which one of them generates the given string.
Can you write Python code for it?

from itertools import permutations
string = 'fbgdceah'
choices = list(permutations('abcdefgh', len(string)))
for password in choices:
    hashed = parse_input(password)
    if hashed == string:
        print('answer', ''.join(password))
        break


I am working on the file "2016/python3/Day21.ipynb".
The first task is:
Reverse
Only the indexes x and y need to be reversed, everything else should be copied or kept as is. We use python's excellent list splicing and inbuilt reverse ([::-1]) to do all the work for us.
Can you write Python code for it?

def reverse(password, x, y):
    x = int(x)
    y = int(y) + 1  # to include character at `y`

#     EXPLICIT
#     _password = []
#     for i in range(0, x):
#         _password.append(password[i])
#     # step in decreasing order
#     for i in range(y, x - 1, -1):
#         _password.append(password[i])
#     for i in range(y + 1, len(password)):
#         _password.append(password[i])
#     return _password

    return password[:x] + password[x:y][::-1] + password[y:]


I am working on the file "2016/python3/Day21.ipynb".
The first task is:
Rotate right and left
We split the string based on the position of x from the front or back based on left/right and then attach it back to the other end.
right --> remove from end, attach to front
left  --> remove from front, attach to end
Can you write Python code for it?

def rotate_right(password, x):
    x = int(x)
    return password[-x:] + password[:-x]

def rotate_left(password, x):
    x = int(x)
    return password[x:] + password[:x]


I am working on the file "2016/recommendation-systems/Exercise.ipynb".
The first task is:
Algoritmo de Recomendação
A primeira etapa da implementação do nosso algoritmo envolve essa função auxiliar, definida abaixo. Ela deve ser implementada de acordo com a implementação do algoritmo abaixo.
Can you write Python code for it?

# ALGORITMO: most_similar_users_to
# ENTRADA: usuario_id: identificador de um usuário. O índice do usuário na matriz dos dados.
# SAÍDA: Lista de 2-tuplas contendo ( outro usuário != user_id, similaridade entre o usuário e user_id).
#        A lista retornada deve ser ordenada pela similaridade entre os usuários.
#
# BEGIN
#
# pares <- []
#
# FOR EACH outro_usuario_id, similaridade IN enumerate(user_similarities) THEN:
#   IF outro_usuario_id != user_id AND similaridade > 0 THEN:
#     pares.add((outro_usuario_id, similaridade))
#
# RETURN ordenar(pares, por=similaridade, em_ordem=decrescente)
#
# END

def most_similar_users_to(user_id):
    return []


I am working on the file "2016/recommendation-systems/Exercise.ipynb".
The first task is:
Função de Similaridade
Implemente a função de similaridade cosseno. Essa função retorna valores entre 0 e 1. Quanto mais próximo de 1, mais semelhantes são os vetores. Quanto mais próximo de 0, mais diferentes eles são.
Can you write Python code for it?

def cosine_similarity(va, vb):
    return 0.0


I am working on the file "2016/recommendation-systems/Exercise.ipynb".
The first task is:
O código abaixo cria um vetor global de todos os possíveis interesses.
Can you write Python code for it?

unique_interests = sorted(list({ interest
                                for user_interests in users_interests
                                for interest in user_interests }))
unique_interests


I am working on the file "2016/recommendation-systems/Exercise.ipynb".
The first task is:
Usando a função de similaridade implementada, o código abaixo cria uma outra matriz. Essa nova matriz é NxN, onde N é o número de usuários. Cada célula (i, j) contém a similaridade do vetor de interesse do usuário i com o vetor de interesse do usuário j.
Can you write Python code for it?

user_similarities = [[cosine_similarity(user_i_interest_vector, user_j_interest_vector)
                      for user_j_interest_vector in user_interest_matrix]
                     for user_i_interest_vector in user_interest_matrix]

eu.print_matrix(user_similarities)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20151008_OpenGeo_Reuse_under_licence-checkpoint.ipynb".
The first task is:
The environment
In order to replicate my environment you need to know what I have installed!
Set up watermark
This describes the versions of software used during the creation.
Please note that critical libraries can also be watermarked as follows:
python
%watermark -v -m -p numpy,scipy
Can you write Python code for it?

%install_ext https://raw.githubusercontent.com/rasbt/python_reference/master/ipython_magic/watermark.py
%load_ext watermark

%watermark -a "Anthony Beck"  -d -v -m -g

#List of installed conda packages
!conda list

#List of installed pip packages
!pip list


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20151008_OpenGeo_Reuse_under_licence-checkpoint.ipynb".
The first task is:
To close this instances press control 'c' in the ipython notebook terminal console
Static presentations allow the presenter to see speakers notes (use the 's' key)
If running dynamically run the scripts below
Pre load some useful libraries
Can you write Python code for it?

#Future proof python 2
from __future__ import print_function #For python3 print syntax
from __future__ import division

# def
import IPython.core.display

# A function to collect user input - ipynb_input(varname='username', prompt='What is your username')

def ipynb_input(varname, prompt=''):
        """Prompt user for input and assign string val to given variable name."""
        js_code = ("""
            var value = prompt("{prompt}","");
            var py_code = "{varname} = '" + value + "'";
            IPython.notebook.kernel.execute(py_code);
        """).format(prompt=prompt, varname=varname)
        return IPython.core.display.Javascript(js_code)

# inline

%pylab inline


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20151008_OpenGeo_Reuse_under_licence-checkpoint.ipynb".
The first task is:
\newpage
About me

Research Fellow, University of Nottingham: orcid
Director, Geolytics Limited - A spatial data analytics consultancy

About this presentation

Available on GitHub - https://github.com/AntArch/Presentations_Github/
Fully referenced PDF

\newpage
A potted history of mapping
In the beginning was the geoword
and the word was cartography

\newpage

Cartography was king. Static representations of spatial knowledge with the cartographer deciding what to represent.
\newpage
And then there was data .........

\newpage

Restrictive data
\newpage

Making data interoperable and open
\newpage
Technical interoperability - levelling the field

\newpage
Facilitating data driven visualization

From Map to Model The changing paradigm of map creation from cartography to data driven visualization
\newpage

\newpage

\newpage

\newpage
So where are these new data products?
\newpage
Non-technical interoperability issues?

Issues surrounding non-technical interoperability include:

Policy interoperabilty
Licence interoperability
Legal interoperability
Social interoperability

We will focus on licence interoperability
\newpage

There is a multitude of formal and informal data.
\newpage
What is a licence?
Wikipedia state:

A license may be granted by a party ("licensor") to another party ("licensee") as an element of an agreement between those parties.
A shorthand definition of a license is "an authorization (by the licensor) to use the licensed material (by the licensee)."

Each of these data objects can be licenced in a different way. This shows some of the licences described by the RDFLicence ontology
\newpage

Concepts (derived from Formal Concept Analysis) surrounding licences
\newpage
Two lead organisations have developed legal frameworks for content licensing:

Creative Commons (CC) and
Open Data Commons (ODC).

Until the release of CC version 4, published in November 2013, the CC licence did not cover data. Between them, CC and ODC licences can cover all forms of digital work.

There are many other...
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('jUzGF401vLc')


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20151008_OpenGeo_Reuse_under_licence-checkpoint.ipynb".
The first task is:
\newpage
Here's something I prepared earlier
A live presentation (for those who weren't at the event).....
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('tkRB5Rp1_W4')


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20151008_OpenGeo_Reuse_under_licence-checkpoint_conflict-20151001-162455.ipynb".
The first task is:
Running dynamic presentations
You need to install the RISE Ipython Library from Damián Avila for dynamic presentations
To convert and run this as a static presentation run the following command:
Can you write Python code for it?

!ipython nbconvert 20151008_OpenGeo_Reuse_under_licence.ipynb --to slides --post serve


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20151008_OpenGeo_Reuse_under_licence-checkpoint_conflict-20151001-162455.ipynb".
The first task is:
\newpage
About me

Research Fellow, University of Nottingham: orcid
Director, Geolytics Limited - A spatial data analytics consultancy

About this presentation

Available on GitHub - https://github.com/AntArch/Presentations_Github/
Fully referenced PDF

\newpage
A potted history of mapping
In the beginning was the geoword
and the word was cartography

\newpage

Cartography was king. Static representations of spatial knowledge with the cartographer deciding what to represent.
\newpage
And then there was data .........

\newpage

Restrictive data
\newpage

Making data interoperable and open
\newpage
Technical interoperability - levelling the field

\newpage
Facilitating data driven visualization

From Map to Model The changing paradigm of map creation from cartography to data driven visualization
\newpage

\newpage

\newpage

\newpage
What about non-technical interoperability issues?

Issues surrounding non-technical interoperability include:

Policy interoperabilty
Licence interoperability
Legal interoperability
Social interoperability

We will focus on licence interoperability
\newpage

There is a multitude of formal and informal data.
\newpage

Each of these data objects can be licenced in a different way. This shows some of the licences described by the RDFLicence ontology
\newpage
What is a licence?
Wikipedia state:

A license may be granted by a party ("licensor") to another party ("licensee") as an element of an agreement between those parties.
A shorthand definition of a license is "an authorization (by the licensor) to use the licensed material (by the licensee)."

Concepts (derived from Formal Concept Analysis) surrounding licences
\newpage
Two lead organisations have developed legal frameworks for content licensing:

Creative Commons (CC) and
Open Data Commons (ODC).

Until the release of CC version 4, published in November 2013, the CC licence did not cover data. Between them, CC and ODC licences can cover all forms of digital work.

There are many other licence types
Many are bespoke
Bespoke...
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
Concepts (derived from Formal Concept Analysis) surrounding licences
\newpage
Two lead organisations have developed legal frameworks for content licensing:

Creative Commons (CC) and
Open Data Commons (ODC).

Until the release of CC version 4, published in November 2013, the CC licence did not cover data. Between them, CC and ODC licences can cover all forms of digital work.

There are many other licence types
Many are bespoke
Bespoke licences are difficult to manage
Many legacy datasets have bespoke licences

I'll describe CC in more detail
\newpage
Creative Commons Zero
Creative Commons Zero (CC0) is essentially public domain which allows:

Reproduction
Distribution
Derivations

\newpage
Constraints on CC0
The following clauses constrain CC0:

Permissions
ND – No derivatives: the licensee can not derive new content from the resource.

Requirements
BY – By attribution: the licensee must attribute the source.
SA – Share-alike: if the licensee adapts the resource, it must be released under the same licence.

Prohibitions
NC – Non commercial: the licensee must not use the work commercially without prior approval.

CC license combinations
License|Reproduction|Distribution|Derivation|BY|SA|NC
----|----|----|----|----|----|----
CC0|X|X|X|||
CC-BY-ND|X|X||X||
CC-BY-NC-ND|X|X||X||X
CC-BY|X|X|X|X||
CC-BY-SA|X|X|X|X|X|
CC-BY-NC|X|X|X|X||X
CC-BY-NC-SA|X|X|X|X|X|X
Table: Creative Commons license combinations
\newpage
Why are licenses important?

They tell you what you can and can't do with 'stuff'
Very significant when multiple datasets are combined
It then becomes an issue of license compatibility

\newpage
Which is important when we mash up data
Certain licences when combined:

Are incompatible
Creating data islands

Inhibit commercial exploitation (NC)
Force the adoption of certain licences
If you want people to commercially exploit your stuff don't incorporate CC-BY-NC-SA data!

Stops the derivation of new works

\newpage

\newpage

A conceptual licence processing workflow....
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
Linked Data and Geo
\newpage
GeoSPARQL employs spatial calculus

\newpage
Querying Linked Data in the wild
The Ordnance Survey
A URI for every place in the UK
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://data.ordnancesurvey.co.uk/doc/50kGazetteer/177276', width=1000, height=700)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
Running dynamic presentations
You need to install the RISE Ipython Library from Damián Avila for dynamic presentations
To convert and run this as a static presentation run the following command:
Can you write Python code for it?

# Notes don't show in a python3 environment

!ipython nbconvert 20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb --to slides --post serve


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
The environment
In order to replicate my environment you need to know what I have installed!
Set up watermark
This describes the versions of software used during the creation.
Please note that critical libraries can also be watermarked as follows:
python
%watermark -v -m -p numpy,scipy
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
To close this instances press control 'c' in the ipython notebook terminal console
Static presentations allow the presenter to see speakers notes (use the 's' key)
If running dynamically run the scripts below
Pre load some useful libraries
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
\newpage
A more robust logic

Would need to decouple licence incompatibility from licence name into licence clause (see table below)
Deal with all licence type
Provide recommendations based on desired derivative licence type
Link this through to the type of process in a workflow:
data derivation is, from a licence position, very different to contextual display

License|Reproduction|Distribution|Derivation|BY|SA|NC
----|----|----|----|----|----|----
CC0|X|X|X|||
CC-BY-ND|X|X||X||
CC-BY-NC-ND|X|X||X||X
CC-BY|X|X|X|X||
CC-BY-SA|X|X|X|X|X|
CC-BY-NC|X|X|X|X||X
CC-BY-NC-SA|X|X|X|X|X|X
ODC-PDDL|X|X|X|||
ODC-BY|X|X|X|X||
ODC-ODbL|X|X|X|X|X|
OGL 2.0|X|X|X|X||
OS OpenData|X|X|X|X|?|
Table: Creative Commons license combinations
\newpage
OGC and Licence interoperability

The geo business landscape is increasingly based on integrating heterogeneous data to develop new products
Licence heterogeneity is a barrier to data integration and interoperability
A licence calculus can help resolve and identify heterogenties leading to
legal compliance
confidence

Use of standards and collaboration with organisations is crucial
Open Data Licensing ontology
The Open Data Institute

Failure to do this could lead to breaches in data licenses
and we all know where that puts us........

\newpage
Linked data and the Semantic Web
The web of Documents

a global filesystem
Designed for human consumption
Primary objects are documents
Expresses links between documents (or sub-parts of)
Degree of structure in objects is fairly low
Semantics of content and links is implicit

The web of Linked Data

a global database
Designed for machines first, humans later
Primary objects are things (or descriptions of things)
Expresses links between things
Degree of structure in (descriptions of) things is high
Semantics of content and links explicit

\newpage
Linked Data
a way of publishing data on the Web that:

encourages reuse
reduces redundancy
maximises its (real and potential)...
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://lov.okfn.org/dataset/lov/', width=1000, height=700)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
\newpage
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://browser.linkedgeodata.org/', width=1000, height=700)


from IPython.display import IFrame
IFrame('http://data.ordnancesurvey.co.uk/id/postcodeunit/NG72QL', width=1000, height=700)


from IPython.display import IFrame
IFrame('http://www.geonames.org/maps/google_52.94_358.8.html', width=1000, height=700)


from IPython.display import IFrame
IFrame('http://data.ordnancesurvey.co.uk/ontology/', width=1000, height=700)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
\newpage
Geonames
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://www.geonames.org/ontology/documentation.html', width=1000, height=700)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
\newpage
It's about re-use
Ontology

An ontology is a shared formal explicit specialisation of a conceptualisation

\newpage
Ontology

The term originated from a philosophy
which deals with the nature and organization of reality

It tries to answer the questions:
What is being?
What are the features common to all beings?
How should things be classified?

Ontology

An ontology is a shared formal explicit specialisation of a conceptualisation

After Agarwal -(@agarwal_ontological_2005):

conceptualisation is identifying relevant abstracted concepts of a phenomena suited to a specific domain
explicit means that the concepts are explicitly defined
formal refers to the fact that the ontology should be machine-readable
shared refers to notion that on ontology captures consensual knowledge

\newpage
Ontology Example

A ‘Carnivore’ is a concept whose members are exactly those animals who eat only meat
A ‘Bear’ is a concept whose members are a kind of ‘Carnivore’
A ‘Cub’ is a concept whose members are exactly those ‘Bear’ whose age is less than one year
A Panda is a individual of a ‘Bear’

We can use these concepts to infer new information from facts.
For example: from the fact 'Ching Ching' is a newborn Panda we know:
'Ching Ching' is a Panda.
'Ching Ching' is a newborn.
We can infer:
'Ching Ching' is a Bear.
'Ching Ching' is a Carnivore.  ????
'Ching Ching' eats only meat.  ????
If we had other logic that told us that 'newborn' is the same as saying less than one year then we can also infer
'Ching Ching' is a Cub.
In an ontology/RDF you can say Anything about Anything. Whilst carnivore is a generally useful concept about bears it is not specifically useful when considering pandas. The domain of application is clearly important.
\newpage
SPARQL the SQL of the semantic web
Find me the capital of all countries in Africa:
PREFIX abc: <nul://sparql/exampleOntology#> .
SELECT ?capital ?country
WHERE {
  ?x abc:cityname ?capital ;
     abc:isCapitalOf ?y.
  ?y abc:countryname ?country ;
     abc:isInContinent...
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://www.opengeospatial.org/projects/groups/geosparqlswg', width=1000, height=700)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
\newpage
So...... we have access to lots of data and software

Formal and Informal
Open and Proprietary

Where are these new data products?
Data, data everywhere - but where are the new derivatives and services?
\newpage
Interoperability
The Defense domain are a bit more explicit......

As defined by DoD policy, interoperability is the ability of systems, units, or forces to provide data, information, material, and services to, and accept the same from, other systems, units, or forces; and to use the data, information, material, and services so exchanged to enable them to operate effectively together. IT and NSS interoperability includes both the technical exchange of information and the end-to-end operational effectiveness of that exchanged information as required for mission accomplishment. Interoperability is more than just information exchange; it includes systems, processes, procedures, organizations, and missions over the life cycle and must be balanced with information assurance.

@watson_joint_2010
\newpage
Non-technical interoperability issues?

\newpage
\newpage

\newpage
Non-technical interoperability
Issues surrounding non-technical interoperability include:

Policy interoperabilty
Licence interoperability
Legal interoperability
Social interoperability

We will focus on licence interoperability
\newpage
Policy Interoperability
The relationship between:

Individuals
Organisations
Countries

Policy determines what, who and how different content can be accessed.
In addition to other elements the policy statements determine:

Authentication
Authorization
Audit

See @innocenti_towards_2011 for more details
\newpage
Social (or human) Interoperability
Social interoperability is concerned about the environment and business and human processes.

Tools are used by people
The social dimension of operational use is underestimated (it's difficult)
People form complex inclusive and exclusive networks
These operate at many scales

US Department of Defence researchers have advocated the development of Policy,...
Can you write Python code for it?

### Export this notebook as markdown
commandLineSyntax = 'dot -Tpng FCA_ConceptAnalysis.dot > FCA_ConceptAnalysis.png'
commandLineSyntax = 'dot -Tsvg FCA_ConceptAnalysis.dot > FCA_ConceptAnalysis.svg'
print (commandLineSyntax)

os.system(commandLineSyntax)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/2016_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
To close this instances press control 'c' in the ipython notebook terminal console
Static presentations allow the presenter to see speakers notes (use the 's' key)
If running dynamically run the scripts below
Pre load some useful libraries
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/2016_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
\newpage
About me

Honorary Research Fellow, University of Nottingham: orcid
Director, Geolytics Limited - A spatial data analytics consultancy

About this presentation

Available on GitHub - https://github.com/AntArch/Presentations_Github/
Fully referenced PDF

\newpage
Contribution to GIScience learning outcomes
This presentation contributes to the following learning outcomes for this course.

Knowledge and Understanding:
Appreciate the importance of standards for Geographic Information and the role of the Open Geospatial Consortium.
Understand the term 'interoperability'.
Appreciate the different models for database design.
Understand the basis of Linked Data.
Find UK government open data and understand some of the complexities in the use of this data.
Appreciate the data issues involved in managing large distributed databases, Location-Based Services and the emergence of real-time data gathering through the 'Sensor-Web'.
Understand the different models for creating international Spatial Data Infrastructures.

Intellectual Skills:
Evaluate the role of standards and professional bodies in GIS.
Articulate the meaning and importance of interoperability, semantics and ontologies.
Assess the technical and organisational issues which come into play when attempting to design large distributed geographic databases aimed at supporting 'real-world' problems.

Suggested Questions:

What do you think are the challenges for National Mapping Agencies and major Volunteer Geographic Data initiatives (OSM) are over the next 5 to 10 years?
This questions aims to get the student to think about how technology, policy and practice frameworks will impact upon the GI landscape. Will, for example, OSM make the Ordnance Survey irrelevant? What would then happen to organisations like the Land Registry that require credible data products to support their activites. How will things like the semantic web impact on GI. Can the aspirations of Open Data be realised and bring a wealth of nwe products and services. What impact will the...
Can you write Python code for it?

# THINGS TO DISCUSS

## Data needs to be credible, accessible, properly licenceed. Bring in Internet of Things


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/.ipynb_checkpoints/2016_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData-checkpoint.ipynb".
The first task is:
\newpage
Here's something I prepared earlier
A live presentation (for those who weren't at the event).....
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
Concepts (derived from Formal Concept Analysis) surrounding licences
\newpage
Two lead organisations have developed legal frameworks for content licensing:

Creative Commons (CC) and
Open Data Commons (ODC).

Until the release of CC version 4, published in November 2013, the CC licence did not cover data. Between them, CC and ODC licences can cover all forms of digital work.

There are many other licence types
Many are bespoke
Bespoke licences are difficult to manage
Many legacy datasets have bespoke licences

I'll describe CC in more detail
\newpage
Creative Commons Zero
Creative Commons Zero (CC0) is essentially public domain which allows:

Reproduction
Distribution
Derivations

\newpage
Constraints on CC0
The following clauses constrain CC0:

Permissions
ND – No derivatives: the licensee can not derive new content from the resource.

Requirements
BY – By attribution: the licensee must attribute the source.
SA – Share-alike: if the licensee adapts the resource, it must be released under the same licence.

Prohibitions
NC – Non commercial: the licensee must not use the work commercially without prior approval.

CC license combinations
License|Reproduction|Distribution|Derivation|BY|SA|NC
----|----|----|----|----|----|----
CC0|X|X|X|||
CC-BY-ND|X|X||X||
CC-BY-NC-ND|X|X||X||X
CC-BY|X|X|X|X||
CC-BY-SA|X|X|X|X|X|
CC-BY-NC|X|X|X|X||X
CC-BY-NC-SA|X|X|X|X|X|X
Table: Creative Commons license combinations
\newpage
Why are licenses important?

They tell you what you can and can't do with 'stuff'
Very significant when multiple datasets are combined
It then becomes an issue of license compatibility

\newpage
Which is important when we mash up data
Certain licences when combined:

Are incompatible
Creating data islands

Inhibit commercial exploitation (NC)
Force the adoption of certain licences
If you want people to commercially exploit your stuff don't incorporate CC-BY-NC-SA data!

Stops the derivation of new works

\newpage

\newpage

A conceptual licence processing workflow....
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
Linked Data and Geo
\newpage
GeoSPARQL employs spatial calculus

\newpage
Querying Linked Data in the wild
The Ordnance Survey
A URI for every place in the UK
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
Running dynamic presentations
You need to install the RISE Ipython Library from Damián Avila for dynamic presentations
To convert and run this as a static presentation run the following command:
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
The environment
In order to replicate my environment you need to know what I have installed!
Set up watermark
This describes the versions of software used during the creation.
Please note that critical libraries can also be watermarked as follows:
python
%watermark -v -m -p numpy,scipy
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
\newpage
About me

Honorary Research Fellow, University of Nottingham: orcid
Director, Geolytics Limited - A spatial data analytics consultancy

About this presentation

Available on GitHub - https://github.com/AntArch/Presentations_Github/
Fully referenced PDF

\newpage
Contribution to GIScience learning outcomes
This presentation contributes to the following learning outcomes for this course.

Knowledge and Understanding:
Appreciate the importance of standards for Geographic Information and the role of the Open Geospatial Consortium.
Understand the term 'interoperability'.
Appreciate the different models for database design.
Understand the basis of Linked Data.
Find UK government open data and understand some of the complexities in the use of this data.
Appreciate the data issues involved in managing large distributed databases, Location-Based Services and the emergence of real-time data gathering through the 'Sensor-Web'.
Understand the different models for creating international Spatial Data Infrastructures.

Intellectual Skills:
Evaluate the role of standards and professional bodies in GIS.
Articulate the meaning and importance of interoperability, semantics and ontologies.
Assess the technical and organisational issues which come into play when attempting to design large distributed geographic databases aimed at supporting 'real-world' problems.

A potted history of mapping
In the beginning was the geoword
and the word was cartography

\newpage

Cartography was king.
Static representations of spatial knowledge with the cartographer deciding what to represent.
Hence, maps are domain specific knowledge repositories for spatial data

\newpage
And then there was data .........

\newpage

Restrictive data
\newpage

Disconnected data with different:

Standards
Quality
Databases
Semantics

\newpage
Why is this an issue?
Over to you......

Decision Making
certainty
uncertainty

Co-ordination
Policy formation
Efficiencies
Best Practice

\newpage
INSPIRE
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('xew6qI-6wNk')


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
\newpage
Can you write Python code for it?


from IPython.display import IFrame
IFrame('http://data.ordnancesurvey.co.uk/datasets/code-point-open/explorer/sparql', width=1000, height=700)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
\newpage
Here's something I prepared earlier
A live presentation (for those who weren't at the event).....
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
\newpage
INSPIRE principles

Data should be collected only once and kept where it can be maintained most effectively
It should be possible to combine seamless spatial information from different sources across Europe and share it with many users and applications
It should be possible for information collected at one level/scale to be shared with all levels/scales; detailed for thorough investigations, general for strategic purposes
Geoinformation needed for good governance at all levels should be readily and transparently available
Easy to find what geoinformation is available, how it can be used to meet a particular need, and under which conditions it can be acquired and used

\newpage

Making data interoperable and open
\newpage
Interoperability

is a property of a product or system, whose interfaces are completely understood, to work with other products or systems, present or future, without any restricted access or implementation.

@wikipedia_interoperability_2016
\newpage
Technical interoperability - levelling the field

\newpage
Syntactic Heterogeneity

the difference in data format. The same logical model can be represented in a range of different physical models (for example ESRI shape file or Geography Mark-up Language (GML)).

This mismatch between underlying data models implies that the same information could be represented differently in different organisations.
The most profound difference is in the storage paradigm:

relational,
object orientated or
hybrids.

@beck_uk_2008, @bishr_overcoming_1998
\newpage
Semantic Heterogeneity

Semantic heterogeneity refers to differences in naming conventions and conceptual groupings.

This can be subdivided into naming and cognitive heterogeneities.

Naming (synonym) mismatch arises when semantically identical data items are named differently.
Cognitive (homonym) mismatch arises when semantically different data items are named identically.
Cognitive semantics can be subtle, reflecting the domain of discourse.

@beck_uk_2008,...
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://www.opengeospatial.org/standards', width=1000, height=700)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
\newpage
Interoperability in action

\newpage
What did technical interoperability facilitate

From Map to Model The changing paradigm of map creation from cartography to data driven visualization
\newpage

\newpage
The world was a happy place.......
Our data was interoperable!

\newpage
Then ....... along came open data

\newpage
The Open landscape integrates formal and informal data

\newpage

\newpage
Background  - originally a grass roots (community) movement..

Open access to knowledge gained significant momementum with the increased uptake of the World Wide Web. This is particularly seen in initiatives like Wikipedia (established in 2001) and Open Knowledge (was the Open Knowledge Foundation: established in 2004). Within the Geo community Open Street Map  (also established in 2004) and the Open Source Geospatial Foundation (OSGeo - established in 2006) are key initiatives that promote accessible data and software resources respectively.
Critical to this is that these were grass roots (community) movements that have proven to be highly disruptive to incumbent data providers, practices and policies.
\newpage
Open in government

The impact of these grass roots movements is seen in Open Data (dot) gov. Pioneered by leaders such as Tim Berners Lee and Nigel Shadbolt
The Shakespeare review [-@shakespeare_shakespeare_2013] indicate that the amount of government Open Data, at least in the UK, is only going to grow.
Open data has the potential to trigger a revolution in how governments think about providing services to citizens and how they measure their success: this produces societal impact.
This will require an understanding of citizen needs, behaviours, and mental models, and how to use data to improve services.
\newpage
Valuing Open Data

A McKinsey Global Institute report examines the economic impact of Open Data [@mckinsey_open_2013] and estimates that globally open data could be worth a minimum of $3 trillion annually.
\newpage
Open in academia

Open inquiry is at the heart of the scientific...
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://www.osgeo.org/', width=1200, height=700)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
\newpage
It's about re-use
Ontology

An ontology is a shared formal explicit specialisation of a conceptualisation

\newpage
Ontology

The term originated from a philosophy
which deals with the nature and organization of reality

It tries to answer the questions:
What is being?
What are the features common to all beings?
How should things be classified?

Ontology

An ontology is a shared formal explicit specialisation of a conceptualisation

After Agarwal -(@agarwal_ontological_2005):

conceptualisation is identifying relevant abstracted concepts of a phenomena suited to a specific domain
explicit means that the concepts are explicitly defined
formal refers to the fact that the ontology should be machine-readable
shared refers to notion that on ontology captures consensual knowledge

\newpage
Ontology Example

A ‘Carnivore’ is a concept whose members are exactly those animals who eat only meat
A ‘Bear’ is a concept whose members are a kind of ‘Carnivore’
A ‘Cub’ is a concept whose members are exactly those ‘Bear’ whose age is less than one year
A Panda is a individual of a ‘Bear’

We can use these concepts to infer new information from facts.
For example: from the fact 'Ching Ching' is a newborn Panda we know:
'Ching Ching' is a Panda.
'Ching Ching' is a newborn.
We can infer:
'Ching Ching' is a Bear.
'Ching Ching' is a Carnivore.  ????
'Ching Ching' eats only meat.  ????
If we had other logic that told us that 'newborn' is the same as saying less than one year then we can also infer
'Ching Ching' is a Cub.
In an ontology/RDF you can say Anything about Anything. Whilst carnivore is a generally useful concept about bears it is not specifically useful when considering pandas. The domain of application is clearly important.
\newpage
SPARQL the SQL of the semantic web
Find me the capital of all countries in Africa:
PREFIX abc: <nul://sparql/exampleOntology#> .
SELECT ?capital ?country
WHERE {
  ?x abc:cityname ?capital ;
     abc:isCapitalOf ?y.
  ?y abc:countryname ?country ;
     abc:isInContinent...
Can you write Python code for it?


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData.ipynb".
The first task is:
\newpage
Open Street Map
Can you write Python code for it?

from IPython.display import IFrame
IFrame('http://linkedgeodata.org/About', width=1000, height=700)


I am working on the file "20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData/20160202_Nottingham_GIServices_Lecture3_Beck_InteroperabilitySemanticsAndOpenData_localised.ipynb".
The first task is:
Concepts (derived from Formal Concept Analysis) surrounding licences
\newpage
Two lead organisations have developed legal frameworks for content licensing:

Creative Commons (CC) and
Open Data Commons (ODC).

Until the release of CC version 4, published in November 2013, the CC licence did not cover data. Between them, CC and ODC licences can cover all forms of digital work.

There are many other licence types
Many are bespoke
Bespoke licences are difficult to manage
Many legacy datasets have bespoke licences

I'll describe CC in more detail
\newpage
Creative Commons Zero
Creative Commons Zero (CC0) is essentially public domain which allows:

Reproduction
Distribution
Derivations

\newpage
Constraints on CC0
The following clauses constrain CC0:

Permissions
ND – No derivatives: the licensee can not derive new content from the resource.

Requirements
BY – By attribution: the licensee must attribute the source.
SA – Share-alike: if the licensee adapts the resource, it must be released under the same licence.

Prohibitions
NC – Non commercial: the licensee must not use the work commercially without prior approval.

CC license combinations
License|Reproduction|Distribution|Derivation|BY|SA|NC
----|----|----|----|----|----|----
CC0|X|X|X|||
CC-BY-ND|X|X||X||
CC-BY-NC-ND|X|X||X||X
CC-BY|X|X|X|X||
CC-BY-SA|X|X|X|X|X|
CC-BY-NC|X|X|X|X||X
CC-BY-NC-SA|X|X|X|X|X|X
Table: Creative Commons license combinations
\newpage
Why are licenses important?

They tell you what you can and can't do with 'stuff'
Very significant when multiple datasets are combined
It then becomes an issue of license compatibility

\newpage
Which is important when we mash up data
Certain licences when combined:

Are incompatible
Creating data islands

Inhibit commercial exploitation (NC)
Force the adoption of certain licences
If you want people to commercially exploit your stuff don't incorporate CC-BY-NC-SA data!

Stops the derivation of new works

\newpage

\newpage

A conceptual licence processing workflow....
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('jUzGF401vLc')