markhneedham/blog huggingface dataset:
Puppet: Package Versions - To pin or not to pin
Date: 2013-04-27 13:40:28

Over the last year or so I've spent quite a bit of time working with puppet and one of the things that we had to decide when installing packages was whether or not to specify a particular version.

On the first project I worked on we didn't bother and just let the package manager chose the most recent version.

Therefore if we were installing nginx the puppet code would read like this:

[source,puppet]
----

package { 'nginx':
  ensure  => 'present',
}
----

We can see which version that would install by checking the version table for the package:

[source,bash]
----

$ apt-cache policy nginx
nginx:
  Installed: (none)
  Candidate: 1:1.2.6-1~43~precise1
  Version table:
     1:1.2.6-1~43~precise1 0
        500 http://ppa.launchpad.net/brightbox/ruby-ng/ubuntu/ precise/main amd64 Packages
     1.4.0-1~precise 0
        500 http://nginx.org/packages/ubuntu/ precise/nginx amd64 Packages
     1.1.19-1ubuntu0.1 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise-updates/universe amd64 Packages
     1.1.19-1 0
        500 http://us.archive.ubuntu.com/ubuntu/ precise/universe amd64 Packages
----

In this case if we don't specify a version the Brightbox '1:1.2.6-1~43~precise1' version will be installed.

Running dpkg with the 'compare-versions' flag shows us that this version is considered higher than the nginx.org one:

[source,bash]
----

$ dpkg --compare-versions '1:1.2.6-1~43~precise1' gt '1.4.0-1~precise' ; echo $?
0
----

From what I understand you can http://wiki.debian.org/AptPreferences[pin versions higher up the list] by associating a higher number with them but given that all these versions are set to '500' I'm not sure how it decides on the order!

The problem with not specifying a version is that when a new version becomes available the next time puppet runs it will automatically upgrade the version for us.

Most of the time this isn't a problem but there were a couple of occasions when a version got bumped and something elsewhere stopped working and it took us quite a while to work out what had changed.

The alternative approach is to pin the package installation to a specific version. So if we want the http://nginx.org/[recent 1.4.0 version] installed we'd have the following code:

[source,puppet]
----

package { 'nginx':
  ensure  => '1.4.0-1~precise',
}
----

The nice thing about this approach is that we always know which version is going to be installed.

The problem we now introduce is that when an updated version is added to the repository the old one is typically removed which means a puppet run on a new machine will fail because it can't find the version.

After working with puppet for a few months it becomes quite easy to see when this is the reason for the failure but it creates the perception that 'puppet is always failing' for newer people which isn't so good.

I think on balance I prefer to have the versions explicitly defined because I find it easier to work out what's going on that way but I'm sure there's an equally strong argument for just picking the latest version.
Puppet Debt
Date: 2013-04-16 20:57:53

I've been playing around with a puppet configuration to run a neo4j server on an Ubuntu VM and one thing that has been quite tricky is getting the Sun/Oracle Java JDK to install repeatably.

I adapted Julian's https://github.com/neo4j-contrib/neo4j-puppet/blob/master/manifests/java.pp[Java module] which uses https://github.com/flexiondotorg/oab-java6[OAB-Java] and although it was certainly working cleanly at one stage I somehow ended up with it not working because of failed dependencies:

[source,text]
----

[2013-04-12 07:03:10] Notice: /Stage[main]/Java/Exec[install OAB repo]/returns:  [x] Installing Java build requirements Ofailed
[2013-04-12 07:03:10] Notice: /Stage[main]/Java/Exec[install OAB repo]/returns: ^[[m^O [i] Showing the last 5 lines from the logfile (/root/oab-java.sh.log)...
[2013-04-12 07:03:10] Notice: /Stage[main]/Java/Exec[install OAB repo]/returns:  nginx-common
[2013-04-12 07:03:10] Notice: /Stage[main]/Java/Exec[install OAB repo]/returns:  nginx-extras
[2013-04-12 07:03:10] Notice: /Stage[main]/Java/Exec[install OAB repo]/returns: E: Sub-process /usr/bin/dpkg returned an error code (1)
...
[2013-04-12 07:03:10] Warning: /Stage[main]/Java/Package[sun-java6-jdk]: Skipping because of failed dependencies
[2013-04-12 07:03:10] Notice: /Stage[main]/Java/Exec[default JVM]: Dependency Exec[install OAB repo] has failures: true
[2013-04-12 07:03:10] Warning: /Stage[main]/Java/Exec[default JVM]: Skipping because of failed dependencies
----

I spent a few hours looking at this problem but couldn't quite figure out how to sort out the dependency problem and ended up running part one command manually after which applying puppet again worked.

Obviously this is a bit of a cop out because ideally I'd like it to be possible to spin up the VM in one puppet run without manual intervention.

A couple of days ago I was discussing the problem with https://twitter.com/a5hok[Ashok] and he suggested that it was probably good to know when I could defer fixing the problem to a later stage since having a completely automated spin up isn't my highest priority.

i.e. when I could take on what he referred to as '*Puppet debt*'

I think this is a reasonable way of looking at things and I have worked on projects where we've been baffled by puppet's dependency graph and have setup scripts which run puppet twice until we have time to sort it out.

If we're http://www.markhneedham.com/blog/2013/04/07/treating-servers-as-cattle-not-as-pets/[spinning up new instances frequently] then we have less ability to take on this type of debt because it's going to hurt us much more but if not then I think it is reasonable to defer the problem.

This feels like another type of http://martinfowler.com/bliki/TechnicalDebt.html[technical debt] to me but I'd be interested in others' thoughts and whether I'm just a complete cop out!
neo4j/cypher: Redundant relationships
Date: 2013-04-16 21:41:58

Last week I was writing a query to find the top scorers in the Premier League so far this season alongside the number of games they've played in which initially read like this:

[source,text]
----

START player = node:players('name:*')
MATCH player-[:started|as_sub]-playedLike-[:in]-game-[r?:scored_in]-player
WITH player, COUNT(DISTINCT game) AS games, COLLECT(r) AS allGoals
RETURN player.name, games, LENGTH(allGoals) AS goals
ORDER BY goals DESC
LIMIT 5
----

[source,text]
----

+------------------------------------+
| player.name        | games | goals |
+------------------------------------+
| "Luis Suárez"      | 30    | 22    |
| "Robin Van Persie" | 30    | 19    |
| "Gareth Bale"      | 27    | 17    |
| "Michu"            | 29    | 16    |
| "Demba Ba"         | 28    | 15    |
+------------------------------------+
5 rows
1 ms
----

I modelled whether a player started a game or came on as a substitute with separate relationship types 'started' and 'as_sub' but in this query we're not interested in that, we just want to know whether they played.

In the world of relational database design we tend to try and avoid redundancy but http://www.markhneedham.com/blog/2012/07/21/neo4j-embracing-the-sub-graph/[with graphs this isn't such a big deal] so I thought I may as well add a 'played' relationship whenever a 'as_sub' or 'started' one was being created.

We can then simplify the above query to read like this:

[source,cypher]
----

START player = node:players('name:*')
MATCH player-[:played]-playedLike-[:in]-game-[r?:scored_in]-player
WITH player, COUNT(DISTINCT game) AS games, COLLECT(r) AS allGoals
RETURN player.name, games, LENGTH(allGoals) AS goals
ORDER BY goals DESC
LIMIT 5
----

[source,text]
----

+------------------------------------+
| player.name        | games | goals |
+------------------------------------+
| "Luis Suárez"      | 30    | 22    |
| "Robin Van Persie" | 30    | 19    |
| "Gareth Bale"      | 27    | 17    |
| "Michu"            | 29    | 16    |
| "Demba Ba"         | 28    | 15    |
+------------------------------------+
5 rows
0 ms
----

When I'm querying I often forget that I modelled starting/substitute separately and think the data has screwed up and it's always because I've forgotten to include the 'as_sub' relationship.

Having the 'played' relationship means that no longer happens which is cool.

I have a reasonably small data set so I haven't seen any performance problems from creating this redundancy.

However, since the maximum number of relationships going out from a player would be unlikely to be more than 1000s I don't think it will become one either.

As always I'd be interested in thoughts from others who have come across similar problems or can see something that I've missed.
A/B Testing: Reporting
Date: 2013-04-28 22:32:38

A few months ago I wrote about http://www.markhneedham.com/blog/2013/01/27/ab-testing-thoughts-so-far/[my initial experiences with A/B testing] and since then we've been working on another one and learnt some things around reporting on these types of tests that I thought was interesting.

== Reporting as a first class concern

One thing we changed from our previous test after a suggestion by https://twitter.com/michael_jones[Mike] was to start treating the reporting of data related to the test as a first class citizen.

To do this we created an end point which the main application could send POST requests to in order to record page views and various other information about users.

On our previous test we'd derived the various conversion rates from our main transactional data store but it was really slow and painful because the way we structure data in there is optimised for a completely different use case.

Having just the data we want to report on in a separate data store has massively reduced the time spent generating reports.

However, one thing that we learnt about this approach is that you need to spend some time thinking about what data is going to be needed up front.

If you don't then it will have to be added later on and the reporting on that metric won't cover the whole test duration.

== Drilling down to get insight

In the first test we ran we only really looked at conversion at quite a high level which is good for getting an overview but doesn't give much insight into what's going on.

For this test we started off with higher level metrics but a few days in became curious about what was going on between two of the pages and so created a report that segmented users based on an action they'd taken on the first page.

This allowed us to rule out a theory about a change in conversion which we had initially thought was down to a change we'd made but actually proved to be because of a change in an external factor.

The frustrating part of drilling down into the data is that you don't really know what is it you're going to want to zoom in on so you have to write code for the specific scenario each time!

== Detecting bugs

We generate browser specific metrics on each test that we run and while the conversion rate is generally similar between them there have been some times when there's a big drop in one browser.

More often than not when we've drilled into this we've found that there was actually a Javascript bug that we hadn't detected and we can then go back and sort that out.

An alternative approach would be to have an automated Javascript/Web Driver test suite which ran against each browser. We've effectively traded off the maintenance cost of that for what is usually a small period of inconvenience for some users.
awk: Parsing 'free -m' output to get memory usage/consumption
Date: 2013-04-10 07:03:15

Although I know this problem is already solved by http://collectd.org/[collectd] and http://newrelic.com/[New Relic] I wanted to write a little shell script that showed me the memory usage on a bunch of VMs by parsing the output of +++<cite>+++http://linux.about.com/library/cmd/blcmdl1_free.htm[free]+++</cite>+++.

The output I was playing with looks like this:

[source,text]
----

$ free -m
             total       used       free     shared    buffers     cached
Mem:           365        360          5          0         59         97
-/+ buffers/cache:        203        161
Swap:          767         13        754
----

I wanted to find out what % of the memory on the machine was being used and as I understand it the numbers that we would use to calculate this are the 'total' value on the 'Mem' line and the 'used' value on the 'buffers/cache' line.

I initially thought that the 'used' value I was interested in should be the one on the 'Mem' line but this number includes memory that http://www.linuxatemyram.com/index.html[Linux has borrowed for disk caching] so it isn't the true number.

There's another http://www.linuxatemyram.com/play.html[quite interesting article showing some experiments you can do] to prove this.

So what I wanted to do was get the result of the calculation '203/365' which I wasn't sure how to do until I realised you can match multiple regular expressions with awk like so:

[source,text]
----

$ free -m | awk '/Mem:/ { print $2 } /buffers\/cache/ { print $3 }'
365
203
----

We've now filtered the output down to just our two numbers but another neat thing you can do with awk is change what it uses as its field and record separator.

In this case we want to change the field separator to be the new line character and we'll set the record separator to be nothing because otherwise it defaults to the new line character which will mess with our field separator.

Those two values are set by using http://www.thegeekstuff.com/2010/01/8-powerful-awk-built-in-variables-fs-ofs-rs-ors-nr-nf-filename-fnr/[the 'RS' and 'FS' variables]:

[source,text]
----

$ free -m |
  awk '/Mem:/ { print $2 } /buffers\/cache/ { print $3 }' |
  awk 'BEGIN { RS = "" ; FS = "\n" } { print $2 / $1 }'
0.556164
----

This is still sub optimal because we're using two awk commands rather than one! We can get around that by storing the two memory values in variables and printing them out in an END block:

[source,text]
----

$ free -m |
  awk '/Mem:/ { total=$2 } /buffers\/cache/ { used=$3 } END { print used/total}'
0.556164
----
Treating servers as cattle, not as pets
Date: 2013-04-07 11:41:34

Although I didn't go to http://www.devopsdays.org/events/2013-london/[Dev Ops Days London] earlier in the year I was following the hash tag on twitter and one of my favourites things that I read https://twitter.com/pikesley/statuses/312614061818904576[was the following]:

____
"`Treating servers as cattle, not as pets`" #DevOpsDays
____

I think this is particularly applicable now that a lot of the time we're using virtualised production environments via http://aws.amazon.com/[AWS], http://www.rackspace.co.uk/cloud-servers/[Rackspace] or +++<insert-cloud-provider-here>+++.</p>

At http://www.uswitch.com/[uSwitch] we use AWS and over the last week https://twitter.com/siddharthdawara[Sid] and I spent some time investigating a memory leak by running our applications against two different versions of Ruby.

One of them was from the https://launchpad.net/~brightbox/+archive/ppa[Brightbox repository] and the other was custom built but they had annoyingly different puppet configurations so we decided to treat them as separate machine types.

We spun up one of the custom built Ruby nodes and put it in the load balancer alongside 11 of the other node types and left it for the day serving traffic.

The next day we had look at the http://newrelic.com/[New Relic] memory consumption for both node types and it was clear that the custom built one's memory usage was climbing much more slowly than the other one.

Instead of trying to work out how to change the Ruby version of the 11 existing nodes we realised it would probably be quicker to just spin up 11 new ones with the custom built Ruby and swap them with the existing ones.

This was pretty much as easy as removing the existing nodes from the load balancer and putting the new ones in although we do have one 'special' machine which runs some background jobs.

We needed to make sure there weren't any jobs on its queue that hadn't been processed and then make sure that we tagged one of the new machines so that they could take over that role.

One thing that made it particularly easy for us to do this is that spin up of new VMs is extremely quick and completely automated including the installation and start up of applications.

The only manual step we have is to put the new nodes into the load balancer which I think works ok as a manual step because it gives us a chance to quickly scan the box and check everything spun up correctly.

We install all packages/configuration on nodes using http://docs.puppetlabs.com/man/apply.html[puppet headless] which makes spin up easier than if you use server/client mode where you have to coordinate node registration with the master on spin up.

I do like this philosophy to machines and although I'm sure it doesn't apply to all situations we're almost at the point where if something breaks on a node we might as well spin up a new one while we're investigating and see which finishes first!+++</insert-cloud-provider-here>+++
Sublime: Getting Textmate's Reveal/Select in Side Bar (Cmd + Ctrl + R)
Date: 2013-04-07 01:00:08

After coming across this post about http://delvarworld.github.io/blog/2013/03/16/just-use-sublime-text/[why you should use Sublime Text] I decided to try using it a bit more and one of the things that I missed from Textmate was the way you can select the current file on the sidebar.

In Textmate the shortcut to do that is 'Cmd + Ctrl + R' so I wanted to be able to do something similar or configure Sublime so it responded to the same shortcut.

The option to reveal a file in the side bar is accessible from the context menu by right clicking on the contents of a file after it's opening and selecting 'Reveal in Side Bar' which is a good start.

To map that to a key binding we need to go 'Preferences > Key Bindings (User)' and http://sublimetext.userecho.com/topic/41241-reveal-file-in-project-tree/[put the following into that file]:

[source,text]
----

[
	{ "keys": ["ctrl+super+r"], "command": "reveal_in_side_bar" }
]
----

Of course if we already have other custom key bindings then we can just add it after those instead.

We can work out what the name of commands are by turning on command logging in the Sublime console.

We need to first open the console with 'Ctrl + `" and then http://www.sublimetext.com/forum/viewtopic.php?f=2&t=11217[paste the following]:

[source,text]
----

sublime.log_commands(True)
----

Any commands that we run will now have their name printed in the console window. e.g.

[source,text]
----

>>> sublime.log_commands(True)
command: context_menu {"event": {"button": 1, "x": 390.21484375, "y": 329.66796875}}
command: reveal_in_side_bar
command: rename_path {"paths": ["/Users/markhneedham/code/thinkingingraphs/public/js/bootstrap.js"]}
no command for selector: noop:
command: show_panel {"panel": "console", "toggle": true}
----

We can then setup appropriate key bindings for whichever commands we like.
Python: Reading a JSON file
Date: 2013-04-09 07:23:59

I've been playing around with some code to spin up AWS instances using http://docs.fabfile.org/en/1.6/[Fabric] and https://github.com/boto/boto[Boto] and one thing that I wanted to do was define a bunch of default properties in a JSON file and then load this into a script.

I found it harder to work out how to do this than I expected to so I thought I'd document it for future me!

My JSON file looks like this:

_config/defaults.json_

[source,json]
----

{
	"region" : "eu-west-1",
	"instanceType": "m1.small"
}
----

To read that file we can do the following:

[source,python]
----

>>> open('config/defaults.json').read()
'{\n\t"region" : "eu-west-1",\n\t"instanceType": "m1.small"\n}'
----

We can then use the http://stackoverflow.com/questions/2835559/python-parsing-file-json[json.loads] function to convert that from a string into a Python object:

[source,python]
----

>>> import json
>>> config = json.loads(open('config/defaults.json').read())
>>> config
{u'region': u'eu-west-1', u'instanceType': u'm1.small'}
----

We'd write the following code to get the region:

[source,python]
----

>>> config["region"]
u'eu-west-1'
----

I guess we might want to use a different approach that didn't load the whole string into memory if we had a large JSON file but for my purposes this will do!
Mac OS X: A couple of neat tools
Date: 2013-04-30 20:07:57

When I first started working at http://www.uswitch.com/[uSwitch] https://twitter.com/siddharthdawara[Sid] installed a couple of 'productivity applications' on my Mac which I've found pretty useful but from talking to others I realised they aren't known/being used by everyone.

== Alfred

http://www.alfredapp.com/[Alfred] is a http://en.wikipedia.org/wiki/Quicksilver_(software)[Quick Silver] replacement which allows you to quickly open applications, find files, search Google and more. Even though we're not using half of its features it's still proved to be useful.

I quite like the calculator feature which we've been using for adhoc calculation like working out http://www.markhneedham.com/blog/2013/04/10/awk-parsing-free-m-output-to-get-memory-usageconsumption/[how much free memory there was on a server] or the http://www.markhneedham.com/blog/2013/04/28/ab-testing-reporting/[conversion rate on part of an A/B test].

image::{{<siteurl>}}/uploads/2013/04/calculator.png[Calculator,600]

== Moom

The other application is http://manytricks.com/moom/[Moom] which allows you to move/resize windows.

I didn't see the point when I first saw it but it's actually really useful when you're working on a big monitor and want to put say the terminal alongside the browser.

We have the following shortcuts set up:

image::{{<siteurl>}}/uploads/2013/04/moom1.png[Moom1,474]

That allows us to type 'Ctrl + Space' to make the window fill the left hand side of the screen, 'Alt + Space' to make it fill the right hand side of the screen and 'Alt + Ctrl + Space' to fill the whole screen.

You can also set up shortcuts to allow you to move a window between displays or to rearrange the windows based on certain events.

Highly recommended!

If anyone knows any other cool tools like this I'd love to hear about them.