markhneedham/blog huggingface dataset:
Reading outside your area of interest
Date: 2013-02-25 22:56:11

A reasonable amount of the information that I consume comes either via scanning https://twitter.com/[twitter] or from my http://getprismatic.com/news/home[prismatic] feed but I noticed that I'm quite biased to reading things in similar subject areas.

I tend to end up reading about data mining/science, functional programming and startups and while the articles are mostly interesting it does eventually start to feel like you're in an echo chamber.</li>

I have a subscription to the http://www.acm.org/[ACM] mainly because I enjoy reading the 'Communications of the ACM' magazine which gets sent out every month and until recently I only read articles which I thought would be interesting.

This unfortunately meant that I was adding to the problem I mentioned earlier whereby everything I read is about similar topics.

I decided to try and change that by reading the magazine from cover to cover and although I haven't finished yet it's been an interesting experience and I've read about things that I wouldn't have thought to read about including:

* Do more computer science papers get rejected than those in other subjects?
* How do we allow people to vote in areas where there's been a natural disaster?
* Why are there currently so many post-docs in computer science?
* How should we go about analysing performance problems?

This seems to be a reasonably good way of diversifying what I read/learn because an editor has decided what I should read rather than me choosing or an algorithm choosing based on what I've previously read.

Having said that, I'd be intrigued to know what approaches/strategies others have for getting knowledge of a broader range of topics.
Micro Services: Readme files
Date: 2013-02-25 23:58:51

By my latest count I have around 15 different micro services/applications checked out on my machine which comprise the system that I'm currently working on.

Most of these are Ruby related so it's easy to figure out how to start up a local copy because it's either +++<cite>+++bundle exec rails server+++</cite>+++ if it's a rails application or +++<cite>+++bundle exec backup+++</cite>+++ if it's a sinatra/rack application.

The clojure applications follow a similar convention and we use rake to run any offline tasks.

Despite these conventions there is still a reasonable amount of knowledge around how to actually get these applications to work that remains in people's heads but doesn't necessarily have to.

Although talking to each other is generally a good thing, an exercise that we tried at http://digital.cabinetoffice.gov.uk/category/gds/[GDS] was to try and document in the README file what someone new to an application would need to know to get it working.

They then tried to get it up and running while only referring to the instructions but noted down anything that didn't make sense or any steps which seemed unnecessarily manual and could be automated.

This was almost certainly more frustrating than having someone show you what to do but it scales a bit better and reduces what I think isn't a high value conversation.

I recently came across a blog post Tom Preston Werner wrote a couple of years ago titled 'http://tom.preston-werner.com/2010/08/23/readme-driven-development.html[Readme Driven Development]' in which he goes further in suggesting that we drive out our applications from the README.

____
This document should stand on its own as a testament to your creativity and expressiveness. The Readme should be the single most important document in your codebase; writing it first is the proper thing to do.
____

I don't know if we need to go this far but as we more towards a future where we'll be working with lots of different applications it would be nice not to have to remember everything about them in our head!
Neo4j: The 'thinking in graphs' curve
Date: 2013-11-27 23:09:31

In a couple of Neo4j talks I've done recently I've been asked how long it takes to get used to modelling data in graphs and whether I felt it's simpler than alternative approaches.

My experience of 'thinking in graphs'™ closely mirrors what I believe is a fairly common curve when learning technologies which change the way you think:+++<div align="center">++++++</div>+++

image::{{<siteurl>}}/uploads/2013/11/2013-11-27_23-06-37.png[2013 11 27 23 06 37,600]

There is an initial stage where *it seems really hard because it's different to what we're used to* and at this stage we might want to go back to what we're used to.

If we persevere, however, we will get over that hump and after that *it will be difficult to imagine another way of modelling data* - at least in domains where the connections between different bits of data are important.

Once we're over the hump data modelling should seem like fun and we'll start looking to see whether we can use graphs to solve other problems we've got.

I wasn't sure whether modelling in graphs is simpler than alternative approaches so as a thought experiment I decided to see what part of my football graph would look like if it was modelled in a relational database.

This is the graph model:

image::{{<siteurl>}}/uploads/2013/11/2013-11-26_19-35-28.png[2013 11 26 19 35 28,600]

If we went for a normalised relational design it might look like this:

image::{{<siteurl>}}/uploads/2013/11/rel-football.png[Rel football,520]

I think this is at least equal in complexity to the graph model and it wouldn't be much fun writing queries against it as the joins pile up!

Of course this comparison is on a data set which is very graphy in nature and therefore well suited for Neo4j. There are certainly other domains where a relational approach would be much more effective.
Neo4j: What is a node?
Date: 2013-11-29 19:50:53

One of the first things I needed to learn when I started using Neo4j was how to model my domain using nodes and relationships and it wasn't initially obvious to me what things should be nodes.

Luckily https://twitter.com/iansrobinson[Ian Robinson] showed me a mini-algorithm which I found helpful for getting started. The steps are as follows:

. Write out the questions you want to ask
. Highlight/underline the nouns
. Those are your nodes!

This is reasonably similar to the way that we work out what our objects should be when we're doing OO modelling and I thought I'd give it a try on some of the data sets that I've worked with recently:

* Female *friends* of friends that somebody could go out with
* *Goals* scored by *Arsenal* *players* in a particular *season*
* *Colleagues* who have similar *skills* to me
* *Episodes* of a *TV program* that a particular *actor* appeared in
* *Customers* who would be affected if a piece of *equipment* went in for repair

If you're like me and aren't that great at English grammar we can always cheat and get http://nltk.org/book/ch05.html[NLTK] to help us out:

[source,python]
----

>>> nltk.pos_tag(nltk.word_tokenize("Female friends of friends that somebody could go out with"))
[('Female', 'NNP'), ('friends', 'NNS'), ('of', 'IN'), ('friends', 'NNS'), ('that', 'WDT'), ('somebody', 'NN'), ('could', 'MD'), ('go', 'VB'), ('out', 'RP'), ('with', 'IN')]
----

That tells us the likely tag for each part of speech in the sentence and we can filter the resulting list so we only see nouns like this:

[source,python]
----

>>> nouns = ['NNS', 'NN', 'NP', 'NNP']
>>> [(word, grammar) for (word, grammar) in nltk.pos_tag(nltk.word_tokenize("Female friends of friends that somebody could go out with")) if grammar in nouns]
[('Female', 'NNP'), ('friends', 'NNS'), ('friends', 'NNS'), ('somebody', 'NN')]
----

We can ignore the 'Female' in this sentence (I think it's been picked up as a proper noun because of the capitalisation) which leaves us with 'friends' and 'somebody' In both cases these nouns represent the concept of a person so we'd want to create nodes representing people in this domain.

Let's see how NLTK gets on with our second question:

[source,python]
----

>>> sentence = "Goals scored by Arsenal players in a particular season"
>>> [(word, grammar) for (word, grammar) in nltk.pos_tag(nltk.word_tokenize(sentence)) if grammar in ['NNS', 'NN', 'NP']]
[('Goals', 'NNS'), ('Arsenal', 'NNP'), ('players', 'NNS'), ('season', 'NN')]
----

In this case we'd have goals, teams (e.g. Arsenal), players and seasons as our nodes.

Although this is a very rough algorithm for working out what things should be nodes in a graph I think it's a good way to get started.

After that the other queries we want to write may lead us to change the model to solve our problem even better.
Neo4j: The case of neo4j-shell and the invisible text ft. Windows and the neo4j-desktop 
Date: 2013-11-29 17:08:49

I've been playing around with Neo4j on a Windows VM recently and I wanted to launch http://docs.neo4j.org/chunked/stable/shell.html[neo4j-shell] to run a few queries.

The neo4j-shell script isn't shipped with http://blog.neo4j.org/2013/09/installer-check-desktop-launcher-check.html[Neo4j desktop] which I used to install Neo4j on my VM but we can still launch it from the Windows Command Prompt with the following command:</p ~~~bash C:\Users\Mark> cd "C:\Program Files\Neo4j Community" C:\Program Files\Neo4j Community>jre\bin\java -cp bin\neo4j-desktop-2.0.0-RC1.jar org.neo4j.shell.StartClient Welcome to the Neo4j Shell! Enter 'help' for a list of commands NOTE: Remote Neo4j graph database service 'shell' at port 1337 Want bash-like features? throw in jLine (http://jline.sourceforge.net) on the classpath ~~~

Unfortunately if we start typing in a query we don't get any feedback as to what we're typed, although if we type something syntactically valid and suffix it with a semi colon and press enter it will be evaluated!

e.g. this is the output from entering the following query ~~~cypher START n = node(*) RETURN COUNT(n); ~~~

image::{{<siteurl>}}/uploads/2013/11/2013-11-29_16-53-56.png[2013 11 29 16 53 56,600]

The reason for the problem is to do with the '.shell_history' file that the shell tries to write to the directory it's launched from. In this case we don't have permission to write anything to 'C:\Program Files\Neo4j Community' and so we get no feedback.

To workaround this problem we need to start the command prompt as an administrator which we can do from the apps menu by right clicking the 'Command Prompt' icon:

image::{{<siteurl>}}/uploads/2013/11/2013-11-29_17-00-59.png[2013 11 29 17 00 59,244]

Next we click 'Start as Administrator' and click through the next screen:

image::{{<siteurl>}}/uploads/2013/11/2013-11-29_17-01-14.png[2013 11 29 17 01 14,501]

Now if we launch neo4j-shell and type in our query we get the expected feedback:

image::{{<siteurl>}}/uploads/2013/11/2013-11-29_17-04-26.png[2013 11 29 17 04 26,600]
Python: Generate all combinations of a list
Date: 2013-11-06 07:25:24

Nathan and I have been playing around with different http://scikit-learn.org/stable/auto_examples/[scikit-learn machine learning classifiers] and we wanted to run different combinations of features through each one and work out which gave the best result.

We started with a list of features:

[source,python]
----

all_columns = ["Fare", "Sex", "Pclass", 'Embarked']
----

http://docs.python.org/2/library/itertools.html#itertools.combinations[itertools#combinations] allows us to create combinations with a length of our choice:

[source,python]
----

>>> import itertools as it
>>> list(it.combinations(all_columns, 3))
[('Fare', 'Sex', 'Pclass'), ('Fare', 'Sex', 'Embarked'), ('Fare', 'Pclass', 'Embarked'), ('Sex', 'Pclass', 'Embarked')]
----

We wanted to create combinations of arbitrary length so we wanted to combine a few invocations of that functions like this:

[source,python]
----

>>> list(it.combinations(all_columns, 2)) + list(it.combinations(all_columns, 3))
[('Fare', 'Sex'), ('Fare', 'Pclass'), ('Fare', 'Embarked'), ('Sex', 'Pclass'), ('Sex', 'Embarked'), ('Pclass', 'Embarked'), ('Fare', 'Sex', 'Pclass'), ('Fare', 'Sex', 'Embarked'), ('Fare', 'Pclass', 'Embarked'), ('Sex', 'Pclass', 'Embarked')]
----

If we generify that code to remove the repetition we end up with the following:

[source,python]
----

all_the_features = []
for r in range(1, len(all_columns) + 1):
	all_the_features + list(it.combinations(all_columns, r))

>>> all_the_features
[('Fare',), ('Sex',), ('Pclass',), ('Embarked',), ('Fare', 'Sex'), ('Fare', 'Pclass'), ('Fare', 'Embarked'), ('Sex', 'Pclass'), ('Sex', 'Embarked'), ('Pclass', 'Embarked'), ('Fare', 'Sex', 'Pclass'), ('Fare', 'Sex', 'Embarked'), ('Fare', 'Pclass', 'Embarked'), ('Sex', 'Pclass', 'Embarked'), ('Fare', 'Sex', 'Pclass', 'Embarked')]
----

or if we want to use reduce instead:

[source,python]
----

>>> reduce(lambda acc, x: acc + list(it.combinations(all_columns, x)), range(1, len(all_columns) + 1), [])
[('Fare',), ('Sex',), ('Pclass',), ('Embarked',), ('Fare', 'Sex'), ('Fare', 'Pclass'), ('Fare', 'Embarked'), ('Sex', 'Pclass'), ('Sex', 'Embarked'), ('Pclass', 'Embarked'), ('Fare', 'Sex', 'Pclass'), ('Fare', 'Sex', 'Embarked'), ('Fare', 'Pclass', 'Embarked'), ('Sex', 'Pclass', 'Embarked'), ('Fare', 'Sex', 'Pclass', 'Embarked')]
----

I imagine there is probably a simpler way that I don't know about yet...!
Neo4j: Exploring new data sets with help from Neo4j browser
Date: 2013-10-18 11:43:59

One of the things that I've found difficult when looking at a new Neo4j database is working out the structure of the data it contains.

I'm used to relational databases where you can easily get a list of the table and the foreign keys that allow you to join them to each other.

image::{{<siteurl>}}/uploads/2013/10/2013-10-17_21-27-36.png[2013 10 17 21 27 36,226]

This has traditionally been difficult when using Neo4j but with the http://blog.neo4j.org/2013/10/neo4j-200-m06-introducing-neo4js-browser.html[release of the Neo4j browser] we can now easily get this type of overview by clicking on the Neo4j icon at the top left of the browser.

We'll see something similar to the image on the left which shows the structure of my football graph and we can now discover parts of the graph by clicking on the various labels, properties or relationships.

For example, if we click on the +++<cite>+++Month+++</cite>+++ label the following query will be executed:

image::{{<siteurl>}}/uploads/2013/10/2013-10-18_11-59-49.png[2013 10 18 11 59 49,400]

&nbsp;

That query actually returns all the months in the database since there are only 12 in total, but for other queries we'd get back a subset of the nodes/relationships with that structure.

To explore the structure around the months we might explore which relationships it has by running a query like this:

image::{{<siteurl>}}/uploads/2013/10/2013-10-18_12-18-54.png[2013 10 18 12 18 54,600]

From running this query we learn that there is a +++<cite>+++in_month+++</cite>+++ relationship connecting matches to the month that they were played in.

Another useful exploration mechanism is the +++<cite>+++http://docs.neo4j.org/chunked/milestone/query-match.html#match-all-shortest-paths[allShortestPaths]+++</cite>+++ function which will show us the path between two nodes if one exists. For example the following query would show us the path between Wayne Rooney and March:

image::{{<siteurl>}}/uploads/2013/10/2013-10-18_12-27-12.png[2013 10 18 12 27 12,600]

The text is a bit small but the relationship between Rooney and a match is named +++<cite>+++played_in+++</cite>+++ so if we want to find which matches Rooney played in in March we could easily write that query.

Another neat feature of the browser is that we can save our queries as we go by clicking on the save button and optionally naming the query in a comment:

image::{{<siteurl>}}/uploads/2013/10/2013-10-18_12-37-31.png[2013 10 18 12 37 31,600]

We can then access those queries and re-run them if we want via the star tab on the left hand side:

image::{{<siteurl>}}/uploads/2013/10/2013-10-18_12-34-06.png[2013 10 18 12 34 06,600]

You can get a hold of the browser by downloading milestone 2.0.0-M06 about half way down the http://www.neo4j.org/download[download page] or if you want to hack on the browser it's on https://github.com/neo4j/neo4j-browser[github].
Java: Incrementally read/stream a CSV file
Date: 2013-10-14 07:27:10

I've been doing some work which involves reading in CSV files, for which I've been using http://opencsv.sourceforge.net/[OpenCSV], and my initial approach was to read through the file line by line, parse the contents and save it into a list of maps.

This works when the contents of the file fit into memory but is problematic for larger files where I needed to stream the file and process each line individually rather than all of them after the file was loaded.

I initially wrote a variation on totallylazy's +++<cite>+++https://code.google.com/p/totallylazy/source/browse/src/com/googlecode/totallylazy/Strings.java?spec=svna4a6ac3d443db02821d434a5d6560cb77ec4ef4a&r=a4a6ac3d443db02821d434a5d6560cb77ec4ef4a#55[Strings#lines]+++</cite>+++ to do this and while I was able to stream the file I made a mistake somewhere which meant the number of maps on the heap was always increasing.

After spending a few hours trying to fix this https://twitter.com/mesirii[Michael] suggested that it'd be easier to use an iterator instead and I ended up with the following code:

[source,java]
----

public class ParseCSVFile {
    public static void main(String[] args) throws IOException
    {
        final CSVReader csvReader = new CSVReader( new BufferedReader( new FileReader( "/path/to/file.csv" ) ), '\t' );
        final String[] fields = csvReader.readNext();

        Iterator<Map<String, Object>>() lazilyLoadedFile = return new Iterator<Map<String, Object>>()
        {
            String[] data = csvReader.readNext();

            @Override
            public boolean hasNext()
            {
                return data != null;
            }

            @Override
            public Map<String, Object> next()
            {
                final Map<String, Object> properties = new HashMap<String, Object>();
                for ( int i = 0; i < data.length; i++ )
                {
                    properties.put(fields[i], data[i]);
                }

                try
                {
                    data = csvReader.readNext();
                }
                catch ( IOException e )
                {
                    data = null;
                }

                return properties;
            }

            @Override
            public void remove()
            {
                throw new UnsupportedOperationException();
            }
        };
    }	
}
----

Although this code works it's not the most readable function I've ever written so any suggestions on how to do this in a cleaner way are welcome.
Neo4j: Modelling hyper edges in a property graph
Date: 2013-10-22 22:02:14

At the http://www.meetup.com/graphdb-belgium/[Graph Database meet up in Antwerp] last week we discussed how you would model a hyper edge in a property graph like Neo4j and I realised that I'd done this in my football graph without realising.

A hyper edge is http://www.hyperedge.com.au/why-hyperedge[defined as follows]:

____
A hyperedge is a *connection between two or more vertices*, or nodes, of a hypergraph. A hypergraph is a graph in which generalized edges (called hyperedges) may connect more than two nodes with discrete properties.
____

In Neo4j an edge (or relationship) can only be between itself or another node, there's no way of creating a relationship between more than 2 nodes.

I had problems when trying to model the relationship between a player and a football match because I wanted to say that a player participated in a match and represented a specific team in that match.

I started out with the following model:

image::{{<siteurl>}}/uploads/2013/10/2013-10-22_22-47-13.png[2013 10 22 22 47 13,600]

Unfortunately creating a direct relationship from the player to the match means that there's no way to work out which team they played for.

This information is useful because sometimes players transfer teams in the middle of a season and we want to analyse how they performed for each team.

In a property graph we need to introduce an extra node which links the match, player and team together:

image::{{<siteurl>}}/uploads/2013/10/2013-10-22_22-54-25.png[2013 10 22 22 54 25,600]

Although we are forced to adopt this design it actually helps us *realise an extra entity* in our domain which wasn't visible before - a player's performance in a match.

If we want to capture information about a players' performance in a match we can store it on this node.

We can also easily aggregate players stats by following the +++<cite>+++played+++</cite>+++ relationship without needing to worry about the matches they played in.

The Neo4j manual has a http://docs.neo4j.org/chunked/milestone/cypher-cookbook-hyperedges.html[few more examples of domain models containing hyper edges] which are worth having a look at if you want to learn more.
Git: Commit squashing made even easier using 'git branch --set-upstream'
Date: 2013-07-16 08:13:02

A few days ago I wrote a blog post describing how I wanted to http://www.markhneedham.com/blog/2013/07/13/gitgithub-squashing-all-commits-before-sending-a-pull-request/[squash a series of commits into one bigger one] before making a pull request and in the comments Rob Hunter showed me an even easier way to do so.

To recap, by the end of the post I had the following git config:

[source,bash]
----

$ cat .git/config
[remote "origin"]
	fetch = +refs/heads/*:refs/remotes/origin/*
	url = git@github.com:mneedham/neo4j-shell-tools.git
[branch "master"]
	remote = origin
	merge = refs/heads/master
[remote "base"]
	url = git@github.com:jexp/neo4j-shell-tools.git
	fetch = +refs/heads/*:refs/remotes/base/*
[branch "readme-pull"]
	remote = origin
	merge = refs/heads/readme-pull
[branch "readme"]
	remote = origin
	merge = refs/heads/readme
----

I was working against the remote 'origin' but the actual home of this repository is 'base'.

I'd created a load of commits on 'origin/readme' and had then squashed them all into one commit on 'origin/readme-pull' by using the following command:

[source,bash]
----

$ git rebase -i c4e94f668223d53f6c7364d19aa965d09ea7eb00
----

where 'c4e94f668223d53f6c7364d19aa965d09ea7eb00' is the hash of the last commit that was made in 'base/master'.

Rob suggested that I should try using http://stackoverflow.com/questions/520650/how-do-you-make-an-existing-git-branch-track-a-remote-branch[upstream tracking] to simplify this even further. When we use upstream tracking we create a link between a local and remote repository which in this case is useful for working out where our commits start from.

I thought I'd try it out on another branch. We want to set the new branch to track 'base/master' since that's the one we eventually want to have our commit applied against.

We'll start from the 'readme' branch which has the list of commits that we want to squash

[source,bash]
----

$ git branch
  master
* readme
  readme-pull
----

Now let's create a new branch and then track it against 'base/master':

[source,bash]
----

$ git checkout -b readme-pull-new
Switched to a new branch 'readme-pull-new'
$ git branch --set-upstream readme-pull-new base/master
Branch readme-pull-new set up to track remote branch master from base.
----

Squashing all our commits is now as simple as running the following command:

[source,bash]
----

$ git rebase -i
----

And then choosing 'squash' against all commits except for the first one which can stay as 'pick'. We then need to edit the commit message into shape which mostly involves deleting the commit messages from the commits we've squashed in this instance.

Thanks to Rob for the tip!