markhneedham/blog huggingface dataset:
Apt-Cacher-Server: Extra junk at end of file
Date: 2012-09-07 15:45:16

We've been installing https://help.ubuntu.com/community/Apt-Cacher-Server[Apt-Cache-Server] so that we can cache some of the packages that we're installing using apt-get on our own network.

(Almost) Following the instructions from the home page we added the following to +++<cite>+++/etc/apt/apt.conf.d/01proxy+++</cite>+++:

[source,text]
----

Acquire::http::Proxy "http://apt-cache-server:3142"
----

And when we ran 'apt-get update' we were getting the following error:

[source,text]
----

E: Syntax error /etc/apt/apt.conf.d/01proxy:2: Extra junk at end of file
----

We initially thought it must be a problem with having an extra space or line ending but it turns out we had just left off the semi colon. D'oh!

The following is what we needed:

[source,text]
----

Acquire::http::Proxy "http://apt-cache-server:3142";
----

We also came across http://askubuntu.com/questions/61540/how-to-update-software-through-proxy[a post on StackOverflow] which suggested that sometimes that doesn't work and something more like this is required:

[source,text]
----

Acquire::http { Proxy "http://apt-cache-server:3142"; };
----

Either of those seem to work for us but I guess YMMV.
neo4j/cypher: CREATE UNIQUE - "SyntaxException: string matching regex ``$' expected but ``p' found"
Date: 2012-09-09 22:29:33

I've been playing around with the mutating cypher syntax of neo4j which allows you to make changes to the graph as well as query it, a feature introduced into cypher in May in http://blog.neo4j.org/2012/05/neo4j-18m01-release-vindeln-vy.html[release 1.8 M01].

I was trying to make use of the 'http://docs.neo4j.org/chunked/milestone/query-create-unique.html[CREATE UNIQUE]' syntax which allows you to create nodes/relationships if they're missing but won't do anything if they already exists.

I had something like the following:

[source,text]
----

START product1=node:products('product_id:1')
CREATE UNIQUE product1-[:sold]->(sales200010 {type:"product_sales", value: 1, name: "Oct 2000 Sales"})
----

So I already have a product indexed with a 'product_id' of 1 and I wanted to create a relationship to a node defining the sales for that product.

When I tried to execute that query I was ending up with the following error:

[source,text]
----

SyntaxException: string matching regex `$' expected but `p' found

Think we should have better error message here? Help us by sending this query to cypher@neo4j.org.

Thank you, the Neo4j Team.
----

I was a bit puzzled as to why that wouldn't work but eventually I read the http://blog.neo4j.org/2012/08/neo4j-18m07-sharing-is-caring.html[release notes for 1.8 M07] where I learnt that in earlier versions of neo4j 'CREATE UNIQUE' had actually been known as 'RELATE' instead.

Since I'm using 1.8 M06 at the moment I just needed to change 'CREATE UNIQUE' to 'RELATE' but http://neo4j.org/download-thanks/?edition=community&release=1.8.RC1&platform=unix[RC1 is now available] so I'll soon switch to that and go back to 'CREATE UNIQUE' again.
Data Science: Making sense of the data
Date: 2012-09-30 14:58:11

Over the past month or so https://twitter.com/a5hok[Ashok] and I have been helping one of our clients explore and visualise some of their data and one of the first things we needed to do was make sense of the data that was available.

== Start small

Ashok suggested that we *work with a subset of our eventual data set* so that we could get a feel for the data and quickly see whether what we were planning to do made sense.

Although our data set isn't at 'big data' levels - we're working with hundreds of thousands/millions of data points rather than the terabytes of data I often read about - I think this worked out well for us.

The data we're working with was initially stored in a SQL Server database so the quickest way to get moving was to export it as CSV files and then work with those.

One problem we had with that approach was that we hadn't realised that some product names could have commas in them and since we were using a comma as our field separator this led to products being imported with some quite strange properties!

Since we only had a few thousand records we were able to see that quickly even when running queries which returned all records.

We've been http://www.markhneedham.com/blog/2012/05/05/neo4j-what-question-do-you-want-to-answer/[asking questions of the data] and with the small data set we were able to very quickly get an answer to these questions for a few records and decide whether it would be interesting to find the answer for the whole data set.

== Visual verification of the data

Another useful, and in hindsight obvious, technique is to spend a little bit of time *skimming over the data and look for any patterns which stand out when visualising scanning the file*.

For example, one import that we did had several columns with NULL values in and we initially tried to parse the file and load it into neo4j.

After going back and skimming the file we realised that we hadn't understood how one of the domain concepts worked and those NULL values did actually make sense and we'd need to change our import script.

On another occasion skimming the data made it clear that we'd made a mistake with the way we'd exported the data and we had to go back and try again.

I'm sure there are other useful techniques we can use when first playing around with a new data set so feel free to point those out in the comments!
Testing XML generation with vimdiff
Date: 2012-09-30 15:48:10

A couple of weeks ago I spent a bit of time writing a Ruby DSL to automate the setup of load balancers, firewall and NAT rules through the http://communities.vmware.com/community/vmtn/developer/forums/vcloudapi?rct=j&q=&esrc=s&source=web&cd=1&ved=0CC4QFjAA&url=http://www.vmware.com/go/vcloudapi&ei=QWNoUOz4JqOG0AX74oHwBA&usg=AFQjCNHrnaSOCy4H4jMXwEvIB9WEhp2eXg&sig2=cBNs29bb-q8VoCGf_wN38w[VCloud API].

The VCloud API deals primarily in XML so the DSL is just a thin layer which creates the appropriate mark up.

When we started out we configured everything manually through the web console and then exported the XML so the first thing that the DSL needed to do was create XML that matched what we already had.

My previous experience using testing frameworks to do this is that they'll tell you whether the XML you've generated is equivalent to your expected XML but if they differ *it isn't easy to work out what was different*.

I therefore decided to use a poor man's approach where I first copied one rule into an XML file, attempted to replicate that in the DSL, and then used http://vimdoc.sourceforge.net/htmldoc/diff.html[vimdiff] to compare the files.

Although I had to manually verify whether or not the code was working I found this approach useful as any differences between the two pieces of XML were very easy to see.

90% of the rules were almost identical so I focused on the 10% that were different and once I'd got those working it was reasonably plain sailing.

My vimdiff command read like this:

[source,text]
----

ruby generate_networking_xml.rb > bar && vimdiff -c 'set diffopt+=iwhite' bar initialFirewall.xml
----

After I was reasonably confident that I understood the way that the XML should be generated I created an Rspec test which checked that we could correctly create all of the existing configurations using the DSL.

While discussing this approach with https://twitter.com/jennifersmithco[Jen] she suggested that an alternative would be to start with a Rspec test with most of the rules hard coded in XML and then replace them one by one with the DSL.

I think that probably does make more sense but I still quite like my hacky approach as well!
neo4j: Handling SUM's scientific notation
Date: 2012-09-30 19:47:32

In some of the recent work I've been doing with neo4j the queries I've written have been summing up the values from multiple nodes and after a certain number is reached the value returned used scientific notation.

For example in a cypher query like this:

[source,text]
----

START category = node:categories('category_id:1')
MATCH p = category-[:has_child*1..5]->subCategory-[:has_product]->product-[:sold]->sales
RETURN EXTRACT(n in NODES(p) : n.category_id?),subCategory.category_id, SUM(sales.sales)
----

I might get a result set like this:

[source,text]
----

+------------------------------------------------------------------------------------------------+
| EXTRACT(n in NODES(p) : n.category_id?)        | subCategory.category_id  | SUM(sales.sales)   |
+------------------------------------------------------------------------------------------------+
| ["246","254","255","3279",<null>,<null>]       | "3279"                   | 3.07213e07         |
| ["246","3649","3650","4362",<null>,<null>]     | "4362"                   | 1.023412e06        |
| ["246","287","291","308",<null>,<null>]        | "308"                    | 504712.5999448135  |
+------------------------------------------------------------------------------------------------+
----

I wanted to be able to add the first two rows together but still have them return separately which meant I needed to convert the values into decimal notation in order to do so.

I came across http://stackoverflow.com/questions/8586357/how-to-convert-a-scientific-notation-string-to-decimal-notation[a Stack Overflow thread explaining how to do it in Ruby]:

[source,ruby]
----

> "%f" % "3.07213e07"
=> "30721300.000000"
----

or

[source,ruby]
----

> "3.07213e07".to_f
=> 30721300.0
----

If we want to http://www.coderanch.com/t/417237/java/java/format-scientific-notation-double-non[do the same thing in Java] it'd read like this:

[source,java]
----

double d = Double.parseDouble("3.07213e07");
NumberFormat formatter = new DecimalFormat("###.#####");
String f = formatter.format(d);
System.out.println(f);
// returns 30721300
----

I couldn't see a way to make the SUM function return in decimal notation but it'd be neat if there was a way to.

For now we have to apply some formatting on the result if we want to do any calculations with it.
Data Science: Scrapping the data together
Date: 2012-09-30 13:44:18

On Friday http://martinfowler.com/[Martin], https://twitter.com/drnsmth[Darren] and I were discussing the ThoughtWorks graph that I was working on earlier in the year and Martin pointed out that an interesting aspect of this type of work is that *the data you want to work with isn't easily available*.

You therefore need to find a way to scrap the data together to make some headway and then maybe at a later stage once some progress has been made it will become easier to replace that with a cleaner solution.

In this case I became curious about exploring the relationships between people in ThoughtWorks but there aren't any APIs on our internal systems so I had to find another way to get the data that I wanted.

The obvious way to do that was to get a copy of the database used by our internal staffing system but I didn't know anybody who worked on that team and trying to get the data that way *would therefore be slow and lose my initial enthusiasm*.

My only alternative was to go via our staffing application and derive the data that way.

I ended up writing some Selenium scripts to crawl the application for people, projects and clients and save that data to JSON files which I later parsed to build up the graph.

The other bit of data that I was curious about was the sponsor relationships inside the company which is kept in a Google spreadsheet.

I wasn't allowed access to that spreadsheet until I was able to show what I was going to use the data for so I first needed to put together something using the other data I'd screen scrapped.

Once I did get the spreadsheet I spent around 3 hours cleaning the data so I could integrate it with the other data I had.

This involved fixing misspelt names and updating the spreadsheets where I knew that the data was out of date - it's certainly not very glamorous work but it helped me to http://www.markhneedham.com/blog/2012/06/21/visualising-a-neo4j-graph-using-gephi/[get to a visualisation which I wrote about in an earlier post].

I haven't done a lot of work in this area but I wouldn't be surprised if it's common that we have to use relatively guerilla tactics like the above to get us up and running.
neo4j: The Batch Inserter and the sunk cost fallacy
Date: 2012-09-23 10:29:10

About a year and a half ago http://www.markhneedham.com/blog/2011/04/17/the-sunk-cost-fallacy/[I wrote about the sunk cost fallacy] which is defined like so:

____
*The Misconception*: You make rational decisions based on the future value of objects, investments and experiences. *The Truth*: Your decisions are tainted by the emotional investments you accumulate, and the more you invest in something the harder it becomes to abandon it.
____

Over the past few weeks https://twitter.com/A5HOK[Ashok] and I have been doing some exploration of one of our client's data by modelling it in a neo4j graph and seeing what interesting things the traversals reveal.

We needed to import around 800,000 nodes with ~ 2 million relationships and because I find that the feedback loop in Ruby is much quicker than Java I suggested that we write the data loading code using the https://github.com/andreasronge/neo4j[neo4j.rb] gem.

Initially we just loaded a small subset of the data so that we could get a rough feel for it and check that we were creating relationships between nodes that actually made sense.

It took a couple of minutes to load everything but that was quick enough.

Eventually, however, we wanted to load the full data set and realised that this approach wasn't really going to scale very well.

The first version created every node/relationship within its own transaction and took around an hour to load everything.

To speed that up we batched up the nodes and only committed a transaction every 10,000 nodes which took the time down to around 20 minutes which was not bad but not amazing.

At one stage Ashok suggested we should try out the http://docs.neo4j.org/chunked/stable/batchinsert.html[batch inserter API] but having spent quite a few hours getting the Ruby version into shape *I really didn't want to let it go* - the sunk cost fallacy in full flow!

A couple of days later we got some new data to load on top of the initial graph and Ashok suggested we use the batch inserter just for that bit of data.

Since that didn't involve deleting any of the code we'd already written I was more keen to try that out.

This time we were adding around 200 nodes but another 1 million relationships to the existing nodes and the end to end time for this bit of code to run was 24 seconds.

Having finally been convinced that the batcher inserter was way better than anything else I spent a couple of hours earlier this week moving all our Ruby code over and it now takes just under 2 minutes to load the whole graph.

To learn how to write code for the batcher inserter we followed the examples from https://github.com/neo4j/community/blob/1.7.2/kernel/src/test/java/examples/BatchInsertExampleTest.java[BatchInsertExampleTest] which covered everything that we wanted to do.

Hopefully the next time I come across such a situation I'll be better able to judge when I'm holding onto something even when I should just let go!
Java: Parsing CSV files
Date: 2012-09-23 22:46:09

As I mentioned in a previous post I recently moved a http://www.markhneedham.com/blog/2012/09/23/neo4j-the-batch-inserter-and-the-sunk-cost-fallacy/[bunch of neo4j data loading code from Ruby to Java] and as part of that process I needed to parse some CSV files.

In Ruby I was using http://fastercsv.rubyforge.org/[FasterCSV] which became the http://stackoverflow.com/questions/5011395/what-is-ruby-1-9-standard-csv-library[standard CSV library from Ruby 1.9] but it's been a while since I had to parse CSV files in Java so I wasn't sure which library to use.

I needed a library which could parse a comma separated file where there might be commas in the values of one of the fields. I think that's fairly standard behaviour in any CSV library but my http://www.linuxquestions.org/questions/linux-general-1/parsing-a-comma-separated-csv-file-where-fields-have-commas-in-to-714332/[googling led me to OpenCSV].

It can be http://downloads.sourceforge.net/project/opencsv/opencsv/2.3/opencsv-2.3-src-with-libs.tar.gz?r=&ts=1348439616&use_mirror=ignum[downloaded from here] and so far seems to do the job!

This is an example of how I'm using it:

[source,java]
----

String filePath = "/Users/mneedham/data/awesome-csv-file.csv";
CSVReader reader = new CSVReader(new FileReader(filePath), ',');

List<String[]> csvEntries = reader.readAll();
Iterator<String[]> iterator = csvEntries.iterator();

while (iterator.hasNext()) {
    String[] row = iterator.next();
    System.out.println("field 1: " + row[0]);
}
----

There are more use cases described on the http://opencsv.sourceforge.net/[home page].
Bash: Piping data into a command using heredocs
Date: 2012-09-15 07:54:04

I've been playing around with some data modelled in neo4j recently and one thing I wanted to do is run an adhoc query in the http://docs.neo4j.org/chunked/stable/shell-starting.html[neo4j-shell] and grab the results and do some text manipulation on them.

For example I wrote a query which outputted the following to the screen and I wanted to sum together all the values in the 3rd column:

[source,text]
----

| ["1","2","3"]         | "3"                             | 1234567    |
| ["4","5","6"]         | "6"                             | 8910112    |
----

Initially I was pasting the output into a text file and then running the following sequence of commands to work it out:

[source,text]
----

$ cat blah2.txt| cut -d"|" -f 4  | awk '{s+=$0} END {print s}'
10144679
----

One way to avoid having to create +++<cite>+++blah2.txt+++</cite>+++ would be to echo the output into standard out like so:

[source,text]
----

$ echo "| ["1","2","3"]         | "3"                             | 1234567    |
| ["4","5","6"]         | "6"                             | 8910112    | " | cut -d"|" -f 4  | awk '{s+=$0} END {print s}'
10144679
----

But it gets a bit confusing as the number of lines of results increases and you have to keep copy/pasting the cut and awk parts of the chain around which was annoying.

One of the things I read on the bus this week was a blog post going through a bunch of http://www.catonmat.net/blog/bash-one-liners-explained-part-three/[bash one liners] and half way through it covers piping data into commands using heredocs which I'd completely forgotten about!

A simple example could be to send a simple message to +++<cite>+++cat+++</cite>+++ which will output the message to standard out:

[source,text]
----

$ cat <<EOL
heredoc> hello i am mark
heredoc> EOL
hello i am mark
----

That works if we want to pipe data into a single command but I didn't know how we'd be able to pipe the output of that command to another command.

In fact it's actually reasonably simple:

[source,text]
----

$ cat <<EOL | cut -d"|" -f 4  | awk '{s+=$0} END {print s}'
pipe pipe heredoc> | ["1","2","3"]         | "3"                             | 1234567    |
pipe pipe heredoc> | ["4","5","6"]         | "6"                             | 8910112    |
pipe pipe heredoc> EOL
10144679
----

And now I have no need to create random text files all over my machine!
While waiting for VMs to provision...
Date: 2012-09-12 22:53:39

https://twitter.com/philandstuff[Phil] and I spent part of the day provisioning new virtual machines for some applications that we need to deploy which involves running a provisioning script and then opening another terminal and repeatedly trying to ssh into the box until it succeeds.

Eventually we got bored of doing that so we figured out a nice little one liner to use instead:

[source,text]
----

while :; do ssh 10.0.0.2; done
----

The ':' is a http://urchin.earth.li/~twic/Some_Bash_Scripting_Notes.html[bash noop] and is http://tldp.org/LDP/abs/html/special-chars.html[defined like so]:

____
*null command [colon].* This is the shell equivalent of a "NOP" (no op, a do-nothing operation). It may be considered a synonym for the shell builtin true. The ":" command is itself a Bash builtin, and its exit status is true (0).
____

In this case it helps us to create an infinite loop which exits once an ssh session is established, meaning that the machine has its ssh daemon running and is ready to roll.

Since we're using a puppet client/server setup we also want to run something on the puppet master to make sure that the client's certificate has been signed.

Here we can use the 'http://en.wikipedia.org/wiki/Watch_(Unix)[watch]' command to help us out:

[source,text]
----

watch "puppet cert list -a | grep new-client-new"
----

So we'll see an empty screen until the client has sent a certificate request that's been picked up by the puppet master and then we'll see it come up.

As usual if you know any cooler ways to do the same things let me know in the comments!
Unix: Caught out by shell significant characters
Date: 2012-09-13 00:17:49

One of the applications that https://twitter.com/philandstuff[Phil] and I were deploying today needed a MySQL server and part of our puppet code to provision that node type runs a command to setup the privileges for a database user.

The unevaluated puppet code reads like this:

[source,text]
----

/usr/bin/mysql -h ${host} -uroot ${rootpassarg} -e "grant all on ${name}.* to ${user}@'${remote_host}' identified by '$password'; flush privileges;"
----

In the application we were deploying that expanded into something like this:

[source,text]
----

/usr/bin/mysql -h localhost -uroot root_pw -e "grant all on db_name.* to db_user@'%' identified by 'awe$ome+password'; flush privileges;"
----

Unfortunately when we ran puppet it was executing without any problems but when we tried to connect to MySQL using 'db_user' with a password of 'awe$ome+password' we kept being denied access.

We tried changing the password to 'bob' to see what would happen, expecting that to fail as well, but were actually able to login so we figured there was something wrong with the password.

Phil suggested echoing the command to see what it was being evaluated to in the shell and once we did that we realised that the password was actually being set to 'awe+password' because the +++<cite>+++$ome+++</cite>+++ bit was being evaluated as a shell variable.

This happens because http://www.howtogeek.com/howto/29980/whats-the-difference-between-single-and-double-quotes-in-the-bash-shell/[shell variables are evaluated] if they are enclosed in "" which is the case here as our whole grant statement is enclosed in "". If variables are enclosed in '' then they won't be evaluated:

[source,text]
----

$ mark="foo"; echo "$mark"
foo
----

[source,text]
----

$ mark="foo"; echo '$mark'
$mark
----

In this case we can therefore switch the '' and "" around to solve the problem:

[source,text]
----

/usr/bin/mysql -h localhost -uroot root_pw -e 'grant all on db_name.* to db_user@"%" identified by awe$ome+password"; flush privileges;'
----