markhneedham/blog huggingface dataset:
InetAddressImpl#lookupAllHostAddr slow/hangs
Date: 2015-03-29 00:31:37

Since I upgraded to Yosemite I've noticed that attempts to resolve localhost on my home network have been taking ages (sometimes over a minute) so I thought I'd try and work out why.

This is what my initial +++<cite>+++/etc/hosts+++</cite>+++ file looked like based on the assumption that my machine's hostname was +++<cite>+++teetotal+++</cite>+++:

[source,bash]
----

$ cat /etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1	localhost
255.255.255.255	broadcasthost
::1             localhost
#fe80::1%lo0	localhost
127.0.0.1	wuqour.local
127.0.0.1       teetotal
----

I setup a little test which replicated the problem:

[source,java]
----

import java.net.InetAddress;
import java.net.UnknownHostException;

public class LocalhostResolution
{
    public static void main( String[] args ) throws UnknownHostException
    {
        long start = System.currentTimeMillis();
        InetAddress localHost = InetAddress.getLocalHost();
        System.out.println(localHost);
        System.out.println(System.currentTimeMillis() - start);
    }
}
----

which has the following output:

[source,text]
----

Exception in thread "main" java.net.UnknownHostException: teetotal-2: teetotal-2: nodename nor servname provided, or not known
	at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
	at LocalhostResolution.main(LocalhostResolution.java:9)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
Caused by: java.net.UnknownHostException: teetotal-2: nodename nor servname provided, or not known
	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
	at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
	at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
	... 6 more
----

Somehow my hostname has changed to +++<cite>+++teetotal-2+++</cite>+++ so I added the following entry to +++<cite>+++/etc/hosts+++</cite>+++:

[source,text]
----

127.0.0.1	teetotal-2
----

Now if we run the program we see this output instead:

[source,text]
----
teetotal-2/127.0.0.1
5011
----

It's still taking 5 seconds to resolve which is much longer than I'd expect. After break pointing through the code it seems like it's trying to do an IPv6 resolution rather than IPv4 so I added an +++<cite>+++/etc/hosts+++</cite>+++ entry for that too: 

[source,text]
----
::1 teetotal-2 
----

Now resolution is much quicker: 

[source,text]
----
teetotal-2/127.0.0.1 6
----

Happy days!
Python: matplotlib hangs and shows nothing (Mac OS X)
Date: 2015-03-26 00:02:54

I've been playing around with some of the http://matplotlib.org/[matplotlib] demos recently and discovered that simply copying one of the examples didn't actually work for me.

I was following the http://matplotlib.org/examples/api/barchart_demo.html[bar chart example] and had the following code:

[source,python]
----

import numpy as np
import matplotlib.pyplot as plt

N = 5
ind = np.arange(N)
fig, ax = plt.subplots()
menMeans = (20, 35, 30, 35, 27)
menStd =   (2, 3, 4, 1, 2)
width = 0.35       # the width of the bars
rects1 = ax.bar(ind, menMeans, width, color='r', yerr=menStd)

plt.show()
----

When I execute this script from the command line it just hangs and I don't see anything at all.

Via a http://stackoverflow.com/questions/21129055/no-plot-window-shows-up-with-matplotlib-pyplot-run-with-enthought-canopy-python[combination of] http://stackoverflow.com/questions/7534453/matplotlib-does-not-show-my-drawings-although-i-call-pyplot-show[different] http://stackoverflow.com/questions/2512225/matplotlib-not-showing-up-in-mac-osx[blog posts] (which all suggested different things!) I ended up with the following variation of imports which seems to do the job:

[source,python]
----

import numpy as np
import matplotlib
matplotlib.use('TkAgg')
import matplotlib.pyplot as plt

N = 5
ind = np.arange(N)
fig, ax = plt.subplots()
menMeans = (20, 35, 30, 35, 27)
menStd =   (2, 3, 4, 1, 2)
width = 0.35       # the width of the bars
rects1 = ax.bar(ind, menMeans, width, color='r', yerr=menStd)

plt.show()
----

If I run this script a Python window pops up and contains the following image which is what I expected to happen in the first place!

image::{{<siteurl>}}/uploads/2015/03/2015-03-25_23-56-08.png[2015 03 25 23 56 08]

The thing to notice is that we've had to change the http://matplotlib.org/faq/usage_faq.html#what-is-a-backend[backend] in order to use matplotlib from the shell:

____
With the TkAgg backend, which uses the Tkinter user interface toolkit, you can use matplotlib from an arbitrary non-gui python shell.
____

Current state: Wishing for ggplot!
Python: Streaming/Appending to a file
Date: 2015-03-09 23:00:56

I've been playing around with Twitter's API (via the https://github.com/tweepy/tweepy[tweepy] library) and due to the rate limiting it imposes I wanted to stream results to a CSV file rather than waiting until my whole program had finished.

I wrote the following program to simulate what I was trying to do:

[source,python]
----

import csv
import time

with open("rows.csv", "a") as file:
    writer = csv.writer(file, delimiter = ",")

    end = time.time() + 10
    while True:
        if time.time() > end:
            break
        else:
            writer.writerow(["mark", "123"])
            time.sleep(1)
----

The program will run for 10 seconds and append one line to 'rows.csv' once a second. Although I have used the 'a' flag in my call to 'open' if I poll that file before the 10 seconds is up it's empty:

[source,bash]
----

$ date && wc -l rows.csv
Mon  9 Mar 2015 22:54:27 GMT
       0 rows.csv

$ date && wc -l rows.csv
Mon  9 Mar 2015 22:54:31 GMT
       0 rows.csv

$ date && wc -l rows.csv
Mon  9 Mar 2015 22:54:34 GMT
       0 rows.csv

$ date && wc -l rows.csv
Mon  9 Mar 2015 22:54:43 GMT
      10 rows.csv
----

I thought the flushing of the file was completely controlled by the with block but lucky for me there's actually a +++<cite>+++https://docs.python.org/2/library/io.html#io.IOBase.flush[flush()]+++</cite>+++ function which allows me to force writes to the file whenever I want.

Here's the new and improved sample program:

[source,python]
----

import csv
import time

with open("rows.csv", "a") as file:
    writer = csv.writer(file, delimiter = ",")

    end = time.time() + 10
    while True:
        if time.time() > end:
            break
        else:
            writer.writerow(["mark", "123"])
            time.sleep(1)
            file.flush()
----

And if we poll the file while the program's running:

[source,python]
----

$ date && wc -l rows.csv
Mon  9 Mar 2015 22:57:36 GMT
      14 rows.csv

$ date && wc -l rows.csv
Mon  9 Mar 2015 22:57:37 GMT
      15 rows.csv

$ date && wc -l rows.csv
Mon  9 Mar 2015 22:57:40 GMT
      18 rows.csv

$ date && wc -l rows.csv
Mon  9 Mar 2015 22:57:45 GMT
      20 rows.csv
----

Much easier than I expected - I ♥ python!
Python: Creating a skewed random discrete distribution
Date: 2015-03-30 22:28:23

I'm planning to write a variant of the http://en.wikipedia.org/wiki/Tf%E2%80%93idf[TF/IDF] algorithm over the HIMYM corpus which weights in favour of term that appear in a medium number of documents and as a prerequisite needed a function that when given a number of documents would return a weighting.

It should return a higher value when a term appears in a medium number of documents i.e. if I pass in 10 I should get back a higher value than 200 as a term that appears in 10 episodes is likely to be more interesting than one which appears in almost every episode.

I went through a few different scipy distributions but none of them did exactly what I want so I ended up writing a sampling function which http://stackoverflow.com/questions/24854965/create-random-numbers-with-left-skewed-probability-distribution[chooses random numbers in a range with different probabilities]:

[source,python]
----

import math
import numpy as np

values = range(1, 209)
probs = [1.0 / 208] * 208

for idx, prob in enumerate(probs):
    if idx > 3 and idx < 20:
        probs[idx] = probs[idx] * (1 + math.log(idx + 1))
    if idx > 20 and idx < 40:
        probs[idx] = probs[idx] * (1 + math.log((40 - idx) + 1))

probs = [p / sum(probs) for p in probs]
sample =  np.random.choice(values, 1000, p=probs)

>>> print sample[:10]
[ 33   9  22 126  54   4  20  17  45  56]
----

Now let's visualise the distribution of this sample by plotting a histogram:

[source,python]
----

import matplotlib
matplotlib.use('TkAgg')
import matplotlib.pyplot as plt

binwidth = 2
plt.hist(sample, bins=np.arange(min(sample), max(sample) + binwidth, binwidth))
plt.xlim([0, max(sample)])
plt.show()
----

image::{{<siteurl>}}/uploads/2015/03/2015-03-30_23-25-05.png[2015 03 30 23 25 05]

It's a bit hacky but it does seem to work in terms of weighting the values correctly. It would be nice if it I could smooth it off a bit better but I'm not sure how at the moment.

One final thing we can do is get the count of any one of the values by using the http://docs.scipy.org/doc/numpy/reference/generated/numpy.bincount.html[bincount] function:

[source,python]
----

>>> print np.bincount(sample)[1]
4

>>> print np.bincount(sample)[10]
16

>>> print np.bincount(sample)[206]
3
----

Now I need to plug this into the rest of my code and see if it actually works!
Python: Equivalent to flatMap for flattening an array of arrays
Date: 2015-03-23 00:45:00

I found myself wanting to flatten an array of arrays while writing some Python code earlier this afternoon and being lazy my first attempt involved building the flattened array manually:

[source,python]
----

episodes = [
    {"id": 1, "topics": [1,2,3]},
    {"id": 2, "topics": [4,5,6]}
]

flattened_episodes = []
for episode in episodes:
    for topic in episode["topics"]:
        flattened_episodes.append({"id": episode["id"], "topic": topic})

for episode in flattened_episodes:
    print episode
----

If we run that we'll see this output:

[source,bash]
----

$ python flatten.py

{'topic': 1, 'id': 1}
{'topic': 2, 'id': 1}
{'topic': 3, 'id': 1}
{'topic': 4, 'id': 2}
{'topic': 5, 'id': 2}
{'topic': 6, 'id': 2}
----

What I was really looking for was the Python equivalent to the http://www.markhneedham.com/blog/2011/07/03/clojure-equivalent-to-scalas-flatmapcs-selectmany/[flatmap function] which I learnt can be achieved in Python http://stackoverflow.com/questions/1077015/python-list-comprehensions-compressing-a-list-of-lists[with a list comprehension] like so:

[source,python]
----

flattened_episodes = [{"id": episode["id"], "topic": topic}
                      for episode in episodes
                      for topic in episode["topics"]]

for episode in flattened_episodes:
    print episode
----

We could also choose to http://naiquevin.github.io/a-look-at-some-of-pythons-useful-itertools.html[use itertools] in which case we'd have the following code:

[source,python]
----

from itertools import chain, imap
flattened_episodes = chain.from_iterable(
                        imap(lambda episode: [{"id": episode["id"], "topic": topic}
                                             for topic in episode["topics"]],
                             episodes))
for episode in flattened_episodes:
    print episode
----

We can then simplify this approach a little by wrapping it up in a 'flatmap' function:

[source,python]
----

def flatmap(f, items):
        return chain.from_iterable(imap(f, items))

flattened_episodes = flatmap(
    lambda episode: [{"id": episode["id"], "topic": topic} for topic in episode["topics"]], episodes)

for episode in flattened_episodes:
    print episode
----

I think the list comprehensions approach still works but I need to look into itertools more - it looks like it could work well for other list operations.
Python: Transforming Twitter datetime string to timestamp (z' is a bad directive in format)
Date: 2015-03-15 22:43:17

I've been playing around with importing Twitter data into Neo4j and since Neo4j can't store dates natively just yet I needed to convert a date string to timestamp.

I started with the following which unfortunately throws an exception:

[source,python]
----

from datetime import datetime
date = "Sat Mar 14 18:43:19 +0000 2015"

>>> datetime.strptime(date, "%a %b %d %H:%M:%S %z %Y")

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_strptime.py", line 317, in _strptime
    (bad_directive, format))
ValueError: 'z' is a bad directive in format '%a %b %d %H:%M:%S %z %Y'
----

https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior[%z is actually a valid option] used to extract the timezone but my googling suggests it not working is one of the idiosyncrasies of strptime.

I eventually came across the +++<cite>+++python-dateutil+++</cite>+++ library, as http://stackoverflow.com/questions/3305413/python-strptime-and-timezones[recommended by Joe Shaw on StackOverflow].

Using that library the problem is suddenly much simpler:

[source,bash]
----

$ pip install python-dateutil
----

[source,python]
----

from dateutil import parser
parsed_date = parser.parse(date)

>>> parsed_date
datetime.datetime(2015, 3, 14, 18, 43, 19, tzinfo=tzutc())
----

To get to a timestamp we can use calendar as I've http://www.markhneedham.com/blog/2014/10/20/python-converting-a-date-string-to-timestamp/[described before]:

[source,python]
----

import calendar
timestamp = calendar.timegm(parser.parse(date).timetuple())

>>> timestamp
1426358599
----
Python: Checking any value in a list exists in a line of text
Date: 2015-03-14 02:52:02

I've been doing some log file analysis to see what cypher queries were being run on a Neo4j instance and I wanted to narrow down the lines I looked at to only contain ones which had mutating operations i.e. those containing the words MERGE, DELETE, SET or CREATE

Here's an example of the text file I was parsing:

[source,bash]
----

$ cat blog.txt
MATCH n RETURN n
MERGE (n:Person {name: "Mark"}) RETURN n
MATCH (n:Person {name: "Mark"}) ON MATCH SET n.counter = 1 RETURN n
----

So I only want lines 2 & 3 to be returned as the first one only returns data and doesn't execute any updates on the graph.

I started off with a very crude way of doing this;

[source,python]
----

with open("blog.txt", "r") as ins:
    for line in ins:
        if "MERGE" in line or "DELETE" in line or "SET" in line or "CREATE" in line:
           print line.strip()
----

A better way of doing this is to use the +++<cite>+++https://docs.python.org/2/library/functions.html#any[any]+++</cite>+++ command and make sure at least one of the words exists in the line:

[source,python]
----

mutating_commands = ["SET", "DELETE", "MERGE", "CREATE"]
with open("blog.txt", "r") as ins:
    for line in ins:
        if any(command in line for command in mutating_commands):
           print line.strip()
----

I thought I might be able to simplify the code even further by using https://docs.python.org/2/library/itertools.html[itertools] but my best attempt so far is less legible than the above:

[source,python]
----

import itertools

commands = ["SET", "CREATE", "MERGE", "DELETE"]
with open("blog.txt", "r") as ins:
    for line in ins:
        if len(list(itertools.ifilter(lambda x: x in line, mutating_commands))) > 0:
            print line.strip()
----

I think I'll go with the 2nd approach for now but if I'm doing something wrong with itertools and it's much easier to use than the example I've shown do correct me!
Python: Forgetting to use enumerate
Date: 2015-03-22 01:28:33

Earlier this evening I found myself writing the equivalent of the following Python code while building a stop list for a topic model\...

[source,python]
----

words = ["mark", "neo4j", "michael"]
word_position = 0
for word in words:
   print word_position, word
   word_position +=1
----

\...which is very foolish given that there's https://docs.python.org/2/library/functions.html#enumerate[already a function] that makes it really easy to grab the position of an item in a list:

[source,python]
----

for word_position, word in enumerate(words):
   print word_position, word
----

Python does make things extremely easy at times - you're welcome future Mark!
R: Markov Chain Wikipedia Example
Date: 2015-04-05 10:07:12

Over the weekend I've been https://github.com/gigasquid/howistart/blob/master/clojure/1/index.md[reading] http://freakonometrics.blog.free.fr/index.php?post/2011/12/20/Basic-on-Markov-Chain-(for-parents)[about] http://en.wikipedia.org/wiki/Markov_chain[Markov Chains] and I thought it'd be an interesting exercise for me to translate Wikipedia's example into R code.

But first a definition:

____
A Markov chain is a random process that undergoes transitions from one state to another on a state space. It is required to possess a property that is usually characterized as "memoryless": the probability distribution of the next state depends only on the current state and not on the sequence of events that preceded it.
____

that 'random process' could be moves in a Monopoly game, the next word in a sentence or, as in Wikipedia's example, the next state of the Stock Market.

The diagram below shows the probabilities of transitioning between the various states:

image::{{<siteurl>}}/uploads/2015/04/800px-Finance_Markov_chain_example_state_space.svg_.png[800px Finance Markov chain example state space svg,399]

e.g. if we're in a Bull Market the probability of the state of the market next week being a Bull Market is 0.9, a Bear Market is 0.075 and a Stagnant Market is 0.025.

We can model the various transition probabilities as a matrix:

[source,r]
----

M = matrix(c(0.9, 0.075, 0.025, 0.15, 0.8, 0.05, 0.25, 0.25, 0.5),
          nrow = 3,
          ncol = 3,
          byrow = TRUE)

> M
     [,1]  [,2]  [,3]
[1,] 0.90 0.075 0.025
[2,] 0.15 0.800 0.050
[3,] 0.25 0.250 0.500
----

Rows/Cols 1-3 are Bull, Bear, Stagnant respectively.

Now let's say we start with a Bear market and want to find the probability of each state in 3 weeks time.

We can do this is by multiplying our probability/transition matrix by itself 3 times and then multiplying the result by a vector representing the initial Bear market state.

[source,R]
----

threeIterations = (M %*% M %*% M)

> threeIterations
> threeIterations
       [,1]    [,2]    [,3]
[1,] 0.7745 0.17875 0.04675
[2,] 0.3575 0.56825 0.07425
[3,] 0.4675 0.37125 0.16125

> c(0,1,0) %*% threeIterations
       [,1]    [,2]    [,3]
[1,] 0.3575 0.56825 0.07425
----

So we have a 56.825% chance of still being in a Bear Market, 35.75% chance that we're now in a Bull Market and only a 7.425% chance of being in a stagnant market.

I found it a bit annoying having to type '%*% M' multiple times but luckily the expm library allows us to apply a Matrix power operation:

[source,r]
----

install.packages("expm")
library(expm)

> M %^% 3
       [,1]    [,2]    [,3]
[1,] 0.7745 0.17875 0.04675
[2,] 0.3575 0.56825 0.07425
[3,] 0.4675 0.37125 0.16125
----

The nice thing about this function is that we can now easily see where the stock market will trend towards over a large number of weeks:

[source,r]
----

> M %^% 100
      [,1]   [,2]   [,3]
[1,] 0.625 0.3125 0.0625
[2,] 0.625 0.3125 0.0625
[3,] 0.625 0.3125 0.0625
----

i.e. 62.5% of weeks we will be in a bull market, 31.25% of weeks will be in a bear market and 6.25% of weeks will be stagnant,
R: dplyr - Error in (list: invalid subscript type 'double'
Date: 2015-04-27 22:34:43

In my continued playing around with R I wanted to find the minimum value for a specified percentile given a data frame representing a cumulative distribution function (CDF).

e.g. imagine we have the following CDF represented in a data frame:

[source,r]
----

library(dplyr)
df = data.frame(score = c(5,7,8,10,12,20), percentile = c(0.05,0.1,0.15,0.20,0.25,0.5))
----

and we want to find the minimum value for the 0.05 percentile. We can use the +++<cite>+++filter+++</cite>+++ function to do so:

[source,r]
----

> (df %>% filter(percentile > 0.05) %>% slice(1))$score
[1] 7
----

Things become more tricky if we want to return multiple percentiles in one go.

My first thought was to create a data frame with one row for each target percentile and then pull in the appropriate row from our original data frame:

[source,r]
----

targetPercentiles = c(0.05, 0.2)
percentilesDf = data.frame(targetPercentile = targetPercentiles)
> percentilesDf %>%
    group_by(targetPercentile) %>%
    mutate(x = (df %>% filter(percentile > targetPercentile) %>% slice(1))$score)

Error in (list(score = c(5, 7, 8, 10, 12, 20), percentile = c(0.05, 0.1,  :
  invalid subscript type 'double'
----

Unfortunately this didn't quite work as I expected - https://twitter.com/tonkouts[Antonios] pointed out that this is probably because we're mixing up two pipelines and dplyr can't figure out what we want to do.

Instead he suggested the following variant which uses the +++<cite>+++https://github.com/hadley/dplyr#do[do]+++</cite>+++ function

[source,r]
----

df = data.frame(score = c(5,7,8,10,12,20), percentile = c(0.05,0.1,0.15,0.20,0.25,0.5))
targetPercentiles = c(0.05, 0.2)

> data.frame(targetPercentile = targetPercentiles) %>%
    group_by(targetPercentile) %>%
    do(df) %>%
    filter(percentile > targetPercentile) %>%
    slice(1) %>%
    select(targetPercentile, score)
Source: local data frame [2 x 2]
Groups: targetPercentile

  targetPercentile score
1             0.05     7
2             0.20    12
----

We can then wrap this up in a function:

[source,r]
----

percentiles = function(df, targetPercentiles) {
  # make sure the percentiles are in order
  df = df %>% arrange(percentile)

  data.frame(targetPercentile = targetPercentiles) %>%
    group_by(targetPercentile) %>%
    do(df) %>%
    filter(percentile > targetPercentile) %>%
    slice(1) %>%
    select(targetPercentile, score)
}
----

which we call like this:

[source,r]
----

df = data.frame(score = c(5,7,8,10,12,20), percentile = c(0.05,0.1,0.15,0.20,0.25,0.5))
> percentiles(df, c(0.08, 0.10, 0.50, 0.80))
Source: local data frame [2 x 2]
Groups: targetPercentile

  targetPercentile score
1             0.08     7
2             0.10     8
----

Note that we don't actually get any rows back for 0.50 or 0.80 since we don't have any entries greater than those percentiles. With a proper CDF we would though so the function does its job.
R: non-numeric argument to binary operator
Date: 2015-04-19 23:08:45

When debugging R code, given my Java background, I often find myself trying to print out the state of variables along with an appropriate piece of text like this:

[source,r]
----

names = c(1,2,3,4,5,6)
> print("names: " + names)
Error in "names: " + names : non-numeric argument to binary operator
----

We might try this next:

[source,r]
----

> print("names: ", names)
[1] "names: "
----

which doesn't actually print the +++<cite>+++names+++</cite>+++ variable - only the first argument to the print function is printed.

We'll find more success with the +++<cite>+++https://stat.ethz.ch/R-manual/R-devel/library/base/html/paste.html[paste]+++</cite>+++ function:

[source,r]
----

> print(paste("names: ", names))
[1] "names:  1" "names:  2" "names:  3" "names:  4" "names:  5" "names:  6"
----

This is an improvement but it repeats the 'names:' prefix multiple times which isn't what we want. Introducing the +++<cite>+++https://stat.ethz.ch/R-manual/R-devel/library/base/html/toString.html[toString]+++</cite>+++ function gets us over the line:

[source,r]
----

> print(paste("names: ", toString(names)))
[1] "names:  1, 2, 3, 4, 5, 6"
----