markhneedham/blog huggingface dataset:
neo4j: java.security.NoSuchAlgorithmException: Algorithm [JKS] of type [KeyStore] from provider [org.bouncycastle.jce.provider.BouncyCastleProvider: name=BC version=1.4]
Date: 2012-07-17 00:02:51

I've spent the last couple of hours moving my neo4j graph from my own machine onto a vanilla CentOS VM and initially tried to run neo using a non Sun version of Java which I installed like so:

[source,text]
----

yum install java
----

This is the version of Java that was installed:

[source,text]
----

$ java -version
java version "1.5.0"
gij (GNU libgcj) version 4.4.6 20120305 (Red Hat 4.4.6-4)
----

When I tried to start neo4j:

[source,text]
----

neo4j/bin/neo4j start
----

I ended up with the following exception in the neo4j log @ +++<cite>+++neo4j/data/log/neo4j.0.0.log+++</cite>+++:

[source,text]
----

Caused by: java.security.NoSuchAlgorithmException: Algorithm [JKS] of type [KeyStore]
from provider [org.bouncycastle.jce.provider.BouncyCastleProvider: name=BC version=1.4] is not found
----

I assumed this algorithm was only being packaged with the Sun JDK (although http://www.bouncycastle.org/java.html[the website suggests otherwise]) so I'd need to replace my version of Java to get it working.

I got the 64 bit '.bin' file from the Oracle download page and then installed it using rpm:

[source,text]
----

rpm -ivh jdk-6u33-linux-x64-rpm.bin
----

I now have this version of Java installed and neo4j is much happier:

[source,text]
----

java -version
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
----

After googling around for a bit I'm still not entirely sure why it wasn't able to work with the original version of Java I installed so if anyone could point me in the direction that'd be useful.
neo4j: Shortest Path with and without cypher
Date: 2012-07-19 19:57:31

I was looking back at some code I wrote a few months ago to query a neo4j database to find the shortest path between two people via the colleagues relationships that exist.

image:{{<siteurl>}}/uploads/2012/05/initial.png[]</img>

The initial code, written using neography, looked like this:

[source,ruby]
----

neo = Neography::Rest.new

start_node = neo.get_node(start_node_id)
destination_node = neo.get_node(destination_node_id)

neo.get_paths(start_node,
              destination_node,
              { "type" => "colleagues" },
              depth = 3,
              algorithm = "shortestPath")
----

The neography code eventually makes a POST request to +++<cite>+++/node/\{start_id}/paths+++</cite>+++ and provides a JSON payload containing the other information about the query.

It works fine and is pretty readable but since I wrote it I've learnt how to write queries using the http://docs.neo4j.org/chunked/stable/cypher-query-lang.html[cypher] query language so I thought it would be interesting to contrast the two bits of code against each other.

This is the equivalent written using cypher:

[source,ruby]
----

query =  "START source=node(#{start_node_id}), destination=node(#{destination_node_id})"
query << "MATCH p = allShortestPaths(source-[r:colleagues*..3]->destination)"
query << "RETURN NODES(p)"

neo.execute_query(query)
----

The amount of lines of code is pretty similar but the thing I like about cypher is that it feels much more declarative and therefore, I think at least, easier to understand.

Having said that, the learning curve for the non-cypher API is a bit easier and it's probably best to start with that to get a feel of what sorts of things you can do with a graph.

When I first started learning cypher I was always forgetting which order to put the different key words and what the syntax was for selecting your starting node.

After a while you do get the hang of it though and it starts to feel like any other programming language.

If we didn't have the node ids of our source and destination nodes but we had two people's names which had been stored in a 'people' Lucene index then we could just as easily find the path between them like this:

[source,ruby]
----

START source=node:people(name="Mark Needham"), destination=node:people(name="Cameron Swords")
MATCH p = allShortestPaths(source-[r:colleagues*..3]->destination)
RETURN NODES(p)
----

I use cypher for pretty much everything I do when reading from the graph and since http://blog.neo4j.org/2012/05/neo4j-18m01-release-vindeln-vy.html[version 1.8 M01] it's been possible to construct queries which mutate the graph as well.
neo4j: Embracing the sub graph
Date: 2012-07-21 22:46:06

In May I wrote a blog post explaining how I'd been designing a neo4j graph by http://www.markhneedham.com/blog/2012/05/05/neo4j-what-question-do-you-want-to-answer/[thinking about what questions I wanted to answer about the data].

In the comments Josh Adell gave me the following advice:

____
The neat things about graphs is that multiple subgraphs can live in the same data-space. +
\... +
Keep your data model rich! Don't be afraid to have as many relationships as you need. The power of graph databases comes from finding surprising results when you have strongly interconnected data.
____

At the time I didn't really understand the advice but I've since updated my graph so that it includes 'colleagues' relationships which can be derived by looking at the projects that people had worked together on.

image::{{<siteurl>}}/uploads/2012/07/v2-a.png[V2 a,600]

When I was showing the graph to http://www.linkedin.com/profile/view?id=15082115&authType=NAME_SEARCH&authToken=t4ys&locale=en_US&srchid=66859460-2706-4d7d-848c-3924569bc30e-0&srchindex=1&srchtotal=19&goback=%2Efps_PBCK_*1_Marc_Hofer_*1_*1_*1_*1_*2_*1_Y_*1_*1_*1_false_1_R_*1_*51_*1_*51_true_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2&pvs=ps&trk=pp_profile_name_link[Marc] he thought it would be quite interesting to see all the people that he hadn't worked with before, a query that's very easy to write when we have these two sub graphs.

We could write the following cypher query to find who Marc hadn't worked with on a specific project:

[source,text]
----

START project=node:projects(name="Project X"), me=node:people(name="Marc Hofer")
MATCH project<-[:worked_on]-neverColleague-[c?:colleagues]->me
WHERE c is NULL
RETURN neverColleague
----

We do an index lookup to find the appropriate project node and to find the node that represent's Marc.

THe MATCH clause starts from the project and then work backwards to the people who have worked on it. We then follow an optional colleagues relationship back to Marc.

The WHERE clause makes sure that we only return people that Marc doesn't have a 'colleagues' relationship with i.e. people Marc hasn't worked with.

Along the same lines we could also find out all the people that Marc has worked with from a specific office:

[source,text]
----

START office=node:offices(name="London - UK South"), me=node:people(name="Marc Hofer")
MATCH office<-[r:member_of]-colleague-[c:colleagues]->me
WHERE (NOT(HAS(r.end_date)))
RETURN colleague, c
----

This is reasonably similar to the previous query except the colleagues relationship is no longer optional since we only want to find people that Marc has worked with.

I'm sure there are other queries we could run but these were a couple that I hadn't thought of before I had multiple sub graphs together on the same overall graph.
London Bus Stops API: Mapping northing/easting values to lat/long
Date: 2012-07-30 22:28:38

I started playing around with the http://www.tfl.gov.uk/businessandpartners/syndication/16493.aspx#17463[TFL Bus stop location and routes API] and one of the annoying things about the data is that it uses http://en.wikipedia.org/wiki/Easting_and_northing[easting/northing] values to describe the location of bus stops rather than lat/longs.

The https://raw.github.com/mneedham/london-buses/master/data/stops.csv[first few lines of the CSV file] look like this:

[source,text]
----

1000,91532,490000266G,WESTMINSTER STN <> / PARLIAMENT SQUARE,530171,179738,177,0K08,0
10001,72689,490013793E,TREVOR CLOSE,515781,174783,78,NB16,0
10002,48461,490000108F,HIGHBURY CORNER,531614,184603,5,C902,0
----

For each of the stops I wanted to convert from the easting/northing value to the equivalent lat/long value but I couldn't find a simple way of doing it in code although I did http://www.uk-postcodes.com/eastingnorthing.php?easting=530171&northing=179738[come across an API that would do it for me].

I wrote the following script to save a new CSV file with all the London bus stops and their lat/long location:

[source,ruby]
----

require 'rubygems'
require 'csv'
require 'open-uri'
require 'json'

data_dir = File.expand_path('data') + '/'
file_name = data_dir + "stops.csv"

stops = CSV.read(file_name).drop(1)

out = CSV.open(data_dir + "stops_with_lat_longs.csv","w")

stops.each do |stop|
  code = stop[1]
  easting = stop[4]
  northing = stop[5]
  url  = "http://www.uk-postcodes.com/eastingnorthing.php?easting=#{easting}&northing=#{northing}"
  location = JSON.parse(open(url).read)

  puts "Processing #{stop[3]}: #{location['lat']}, #{location['lng']}"

  out << [code, location['lat'],location['lng']]
end

out.close
----

I've uploaded the https://github.com/mneedham/london-buses/blob/master/data/stops_with_lat_longs.csv[file with mapping from bus stop code to lat/long] to github as well.

http://blog.poggs.com/2010/09/converting-osgb36-eastingsnorthings-to-wgs84-longitudelatitude-in-ruby/[Peter Hicks has a blog post showing another way of doing this using just Ruby code] but I couldn't get the 'proj4' gem to install and I didn't fancy shaving that yak when I had another solution which worked.
Data visualisation: Is 'interesting' enough?
Date: 2012-07-08 22:45:41

I recently read a blog post by Julian Boot titled 'http://julianboot.com/2012/07/visualisation-without-analysis/[visualisation without analysis is fine]' where he suggests that we can learn things from visualising data in the right way - detailed statistical analysis isn't always necessary.

I thought this was quite an interesting observation because over the past couple of months I've been playing around with ThoughtWorks data and looking at different ways to visualise aspects of the data.

For example the following visualisation shows the strength of colleague relationships between the various ThoughtWorks offices:

image::{{<siteurl>}}/uploads/2012/07/map-2.png[Map 2,600]

We can learn some interesting things from looking at it such as:

* There's a very strong connection between Bangalore and London
* There's also a very strong connection between Porto Alegre and Delhi/Dallas
* If we look a bit closer we can see the connections from China aren't as strong - there's a reasonable link to San Francisco and weaker ones to Australia but not so much to other countries.

If we know more about the domain then we'd know that there are some distributed projects going on between Porto Alegre and the other two places and there are currently quite a lot of people from Bangalore working in London.

Although these observations are quite interesting I'm not sure if they are anything more than that.

In my opinion *the intent of visualisations is to provoke some sort of action* by helping people see something in the data which they couldn't see before.

In Julian's post he refers to a visualisation showing the colours used in movie posters and how there's a trend towards a 50/50 split between orange and blue.

It's really interesting to look at but as Julian points out it's http://www.slashfilm.com/orangeblue-contrast-in-movie-posters/[well known that movie posters make heavy use orange and blue] and presumably at some stage someone has already worked out that these are the two most effective colours.

In my example most people I talked to were able to predict where the strongest connections would be before I showed them the visualisation.

My current thinking is that if a visualisation is only 'interesting' then perhaps I haven't played around with the data enough to find some insight that would actually lead to an outcome/improvement of some sort.

I'm definitely interested in hearing others opinions/experienced in this area though!
ganglia: Importing gmond Python modules
Date: 2012-07-08 21:55:53

My colleague http://www.linkedin.com/pub/shodhan-sheth/2/277/287[Shohdan] and I spent a couple of days last week wiring up various monitoring metrics into ganglia and while most of them come built in, we also http://www.mail-archive.com/ganglia-general@lists.sourceforge.net/msg06458.html[found some] https://github.com/ganglia/gmond_python_modules[python based modules that we wanted to use].

Unfortunately we couldn't find any instructions on github explaining how to set them up but after a bit of trial and error we figured it out.

One of the modules that we wanted to use was https://github.com/ganglia/gmond_python_modules/tree/master/diskstat[diskstat] which provides I/O wait time metrics which we couldn't find in the built in modules.

The first thing we needed to do was enable the Python module which comes built in with ganglia:

/etc/ganglia/gmond.conf

[source,text]
----

modules {
  module {
    name = "core_metrics"
  }
  ...
  module {
    name = "python_module"
    path = "/usr/lib/ganglia/modpython.so"
    params = "/usr/lib/ganglia/python_modules/"
  }
}
----

As part of this we had to point the module at the folder location where we'd be putting the python modules - the 'params' argument.

In this case we put https://github.com/ganglia/gmond_python_modules/blob/master/diskstat/python_modules/diskstat.py[diskstat.py] into that folder.

[source,text]
----

$ ls -alh /usr/lib/ganglia/python_modules/
total 52K
drwxr-xr-x 2 root     root     4.0K 2012-07-06 01:49 .
drwxr-xr-x 3 root     root     4.0K 2012-07-03 08:27 ..
-rw-r--r-- 1 mneedham mneedham  15K 2012-07-03 08:28 diskstat.py
-rw-r--r-- 1 root     root      12K 2012-07-03 09:00 diskstat.pyc
----

We also needed to add the https://github.com/ganglia/gmond_python_modules/blob/master/diskstat/conf.d/diskstat.pyconf[diskstat.pyconf] configuration file so that ganglia knows which metrics to measure. We were having problems with the 'name_match' wildcard property and ended up listing the metrics individually:

/etc/ganglia/conf.d/diskstat.pyconf

[source,text]
----

modules {
  module {
    name = 'diskstat'
    language = 'python'

    param devices {
      value = 'sda1'
    }
  }
}

collection_group {
  collect_every = 30
  time_threshold = 30

  metric {
    name = "diskstat_sda1_io_time"
  }
  ...
}
----

The last step was to edit gmond.conf again to pick up the python configuration files:

/etc/ganglia/gmond.conf

[source,text]
----

include ('/etc/ganglia/conf.d/*.conf')
# add this line
include ('/etc/ganglia/conf.d/*.pyconf')
----

There are a bunch of other modules for different tools in the repository and we've followed the above approach for installation of them as well.
R: Mapping a function over a collection of values
Date: 2012-07-23 23:25:00

I spent a bit of Sunday playing around with R and one thing I wanted to do was map a function over a collection of values and transform each value slightly.

I loaded my data set using the 'Import Dataset' option in http://rstudio.org/[R Studio] (suggested to me by https://twitter.com/roryoung[Rob]) which gets converted to the following function call:

[source,r]
----

> data <-  read.csv("~/data.csv", header=T, encoding="ISO-8859")
> data
  Column1 InterestingColumn
1    Mark             12.50
2    Dave            100.00
3    John          1,231.00
----

_data.csv_

[source,text]
----

Column1, InterestingColumn
Mark, 12.50
Dave, 100.00
John, 1,231.00
----

+++<cite>+++data+++</cite>+++ is a table with the type '3 obs. of 2 variables' in R Studio.

I was only interested in the values in the 2nd column so I selected those like this:

[source,r]
----

> data$InterestingColumn
[1]  12.50     100.00    1,231.00
Levels:  1,231.00  100.00  12.50
----

I wanted to apply a function over each of the numbers and return a new list containing those transformations.

I initially had a look at doing this with a for loop but it didn't turn out to be as easy as I'd expected and I eventually came across the +++<cite>+++http://stat.ethz.ch/R-manual/R-patched/library/base/html/lapply.html[lapply]+++</cite>+++ function which allows you to apply a function over a list or vector.

[source,r]
----

> values <- data$InterestingColumn
> lapply(values, function(x) 5000 - as.numeric(gsub("\\s|,","", x)))
[[1]]
[1] 4987.5

[[2]]
[1] 4900

[[3]]
[1] 3769
----

We define a function which subtracts the value in the column from 5000 since the CSV file contained derived values and I was interested in the original value.

In order to do that subtraction I needed to cast the value from the CSV file to be numeric which first involved stripping out any spaces or commas using http://stackoverflow.com/questions/5992082/how-to-remove-all-whitespace-from-a-string-in-r[gsub] and then casting the string using as.numeric.

If we want to have a table structure then we can use the 'by' function to do a similar thing:

[source,r]
----

> as.table(by(data$InterestingColumn, data$Column1, function(x) 5000 - as.numeric(gsub("\\s|,","", x))))
data$Column1
  Dave   John   Mark
4900.0 3769.0 4987.5
----

I don't know enough R to know how to keep the data in exactly the same structure as we got it so if anyone can point me in the right direction that'd be cool.
neo4j: Graph Global vs Graph Local queries
Date: 2012-07-23 22:23:10

A few weeks ago I did a presentation at the ThoughtWorks EU away day on the graph I've been developing using neo4j and I wanted to show who the most connected people in each of our European offices were.

I started with the following cypher query:

[source,text]
----

START n = node(*)

MATCH n-[r:colleagues*1..2]->c, n-[r2:member_of]->office

WHERE n.type? = 'person'
AND (NOT(HAS(r2.end_date)))
AND office.name = 'London - UK South'
AND (NOT(HAS(c.thoughtquitter)))

RETURN n.name, count(distinct(c)) AS connections, office.name

ORDER BY connections DESC
----

The intention is to find all the people who currently work in the London office and find their 1st and 2nd level connections i.e. people they have worked with or people who have worked with people they have worked with.

That gives a rough idea of their reach in the organisation.

I also wanted to exclude anyone who had left, hence the 'thoughtquitter' check.

When I ran this query it took about 15 minutes to complete which immediately made me realise I was doing something wrong although I wasn't sure exactly what.

I showed the query to Michael Hunger and https://gist.github.com/2983831[he pointed out] that it's a graph global query i.e. it probably touches most of the nodes on the graph even though it doesn't necessarily need to.

Michael suggested restructuring the query to start from the London office and work out from there:

[source,text]
----

START office=node:offices(name='London - UK South')

MATCH office<-[r2:member_of]-n-[r:colleagues*1..2]->c

WHERE n.type? = 'person'
AND (NOT(HAS(r2.end_date)))
AND (NOT(HAS(c.thoughtquitter)))

RETURN n.name, count(distinct(c)) AS connections, office.name

ORDER BY connections DESC
----

Now we are starting at the London office and finding all the people who have been members of it at some stage. We go and find their level 2 colleagues and then do the same filters that we had in the first query.

The query time went down to around 45 seconds, presumably because the number of nodes on the graph that we touch has gone down substantially.

The lesson for me from this is that when doing a graph traversal it makes much more sense to *pick specific starting and/or finishing nodes rather than selecting all of them*.
A rogue "\357\273\277" (UTF-8 byte order mark)
Date: 2012-09-03 06:31:54

We've been loading some data into neo4j from a CSV file - creating one node per row and using the value in the first column as the index lookup for the node.

Unfortunately the index lookup wasn't working for the first row but was for every other row.

By coincidence we started saving each row into a hash map and were then able to see what was going wrong:

[source,ruby]
----

require 'rubygems'
require 'fastercsv'

things = FasterCSV.read("things.csv", :col_sep => "|")

saved_things = {}
things do |row|
  saved_things[row[0]] = row[1]
end

p saved_things
----

This is what we saw when we ran the script:

[source,text]
----

{"\357\273\2771"=>"Thing1", "2" => "Thing2"}
----

A bit of googling suggests that "\357\273\277" http://www.highdots.com/forums/macromedia-dreamweaver/357-273-277-characters-placed-154281.html[represents a UTF-8 byte order mark] which apparently http://en.wikipedia.org/wiki/Byte_order_mark[isn't actually needed anyway]:

____
The Unicode Standard does permit the BOM in UTF-8, but does not require or recommend its use. Byte order has no meaning in UTF-8 so in UTF-8 the BOM serves only to identify a text stream or file as UTF-8.
____

We're not converting the CSV file back into any other format so the following awk command can be used to cleanup it up:

[source,text]
----

awk '{if(NR==1)sub(/^\xef\xbb\xbf/,"");print}' things.csv > things.nobom.csv
----

If we use the hexdump tool we can see that the BOM has been removed:

Before:

[source,text]
----

$ hexdump things.csv
0000000 ef bb bf 31 7c 50 72 69 6d 61 72 69 65 73 0d 0a
...
----

After:

[source,text]
----

hexdump things.nobom.csv
0000000 31 7c 50 72 69 6d 61 72 69 65 73 0d 0a 31 30 7c
----

I was initially curious why Ruby and the hexdump were printing out different values but it's just a case of Ruby showing the Octal version of the BOM as compared to the Hexidecimal version. The values translate like so:

[source,text]
----

Octal | Hexadecimal | Decimal
357   | EF          | 239
273   | BB          | 187
277   | BF          | 191
----
zsh: Don't verify substituted history expansion a.k.a.  disabling histverify
Date: 2012-09-16 13:35:56

I use http://www.zsh.org/[zsh] on my Mac terminal and in general I prefer it to bash but it has an annoying default setting whereby when you try to repeat a command via substituted history expansion it asks you to verify that.

For example let's say by mistake I try to vi into a directory rather than cd'ing into it:

[source,text]
----

vi ~/.oh-my-zsh
----

If I try to cd into the directory by using '!$' to grab the last argument from the previous command it will make me confirm that I want to do this:

[source,text]
----

$ cd !$
$ cd ~/.oh-my-zsh
----

While reading another one of Peter Krumins' blog posts, this time about http://www.catonmat.net/blog/the-definitive-guide-to-bash-command-line-history/[bash command line history], I came to learn that this is because a setting called +++<cite>+++http://wiki.bash-hackers.org/internals/shell_options#histverify[histverify]+++</cite>+++ has been enabled.

____
histverify Allow to review a history substitution result by loading the resulting line into the editing buffer, rather than directly executing it.
____

I found http://stackoverflow.com/questions/11917567/how-to-view-default-zsh-settings-histsize-savehist[a thread on StackOverflow which explains all the zsh settings in more detail] but for my purposes I needed to run the following command to disable +++<cite>+++histverify+++</cite>+++:

[source,text]
----

unsetopt histverify
----

I also put that into my +++<cite>+++~/.zshrc+++</cite>+++ file so it will carry across to any new terminal sessions that I open.