markhneedham/blog huggingface dataset:
neo4j: Creating a custom index with neo4j.rb
Date: 2012-08-05 09:45:08

As I http://www.markhneedham.com/blog/2012/07/30/london-bus-stops-api-mapping-northingeasting-values-to-latlong/[mentioned in my last post] I've been playing around with the http://www.tfl.gov.uk/businessandpartners/syndication/16493.aspx#17463[TFL Bus stop location and routes API] and one thing I wanted to do was load all the bus stops into a neo4j database using the https://github.com/andreasronge/neo4j[neo4j.rb] gem.

I initially populated the database via https://github.com/maxdemarzi/neography/[neography] but it was taking around 20 minutes each run and I figured it'd probably be much quicker to populate it directly rather than using the REST API.

Creating nodes is reasonably simple, and the code to add bus stops looks like this:

[source,ruby]
----

require 'neo4j'

Neo4j::Transaction.run do
  stops_to_add = [ {:name => "Walworth Road", :code => 10001 }]

  stops_to_add.each do |stop|
    node = Neo4j::Node.new(:name => stop[:name], :code => stop[:code], :type => "stop")
    puts "Code: #{stop[:code]}, Stop: #{stop[:name]}"
  end
end
----

I wanted to be able to search for bus stops using cypher so I needed to create an index for each stop to allow me to do that easily.

I initially tried creating a +++<cite>+++Stop+++</cite>+++ class and defining the index in there http://neo4j.rubyforge.org/guides/lucene.html[as suggested in the documentation] but from what I could tell it created an index named after the string representation of the +++<cite>+++Stop+++</cite>+++ object which made it difficult to use in http://docs.neo4j.org/chunked/stable/cypher-query-lang.html[cypher].

Eventually I came across https://github.com/andreasronge/neo4j/wiki/Neo4j%3A%3ACore-Lucene[another page] which explained that I needed to create a 'custom index' if I wanted to be able to reference it by name.

I ended up with the following:

[source,ruby]
----

class StopsIndex
  extend Neo4j::Core::Index::ClassMethods
  include Neo4j::Core::Index

  self.node_indexer do
    index_names :exact => 'stops'
    trigger_on :type => "stop"
  end

  index :code
end
----

As far as I understand this index gets triggered when you're inside a transaction adding a node of type 'stop' which is what I'm doing here.

With the index defined this way it's now possible to look up stops using cypher:

[source,text]
----

START stop = node:stops(code = "10001")
RETURN stop
----

And when later on in the code I wanted to add a 'route' between stops I could look up the stops like so:

[source,ruby]
----

Neo4j::Transaction.run do
  stop1 = StopsIndex.find("code: \"10001\"").first
  stop2 = StopsIndex.find("code: \"10002\"").first				
  Neo4j::Relationship.new(:route, stop1, stop2, { :bus_number => 1 })
end
----

The https://github.com/mneedham/london-buses/blob/master/data/load.rb[full code for this is on github].
puppetdb: Failed to submit 'replace catalog' command for client to PuppetDB at puppetmaster:8081: [500 Server Error]
Date: 2012-08-16 23:31:28

I'm still getting used to the idea of *following the logs* when working out what's going wrong with distributed systems but it worked well when trying to work out why our puppet client which was throwing this error when we ran 'puppet agent -tdv':

[source,text]
----

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace catalog' command for client to PuppetDB at puppetmaster:8081: [500 Server Error]
----

We were seeing the same error in +++<cite>+++/var/log/syslog+++</cite>+++ on the puppet master and a quick look at the process list didn't show that the puppet master or puppetdb services were under a particularly heavy load.

http://serverfault.com/questions/403753/puppet-gives-ssl-error-because-master-is-not-running[Eventually] we ended up looking at the puppetdb logs which showed that it was running out of memory:

+++<cite>+++/var/log/puppetdb/puppetdb.log+++</cite>+++

[source,text]
----

2012-08-15 17:48:38,535 WARN  [qtp814855969-66] [server.AbstractHttpConnection] /commands
java.lang.OutOfMemoryError: Java heap space
----

The default +++<cite>+++/etc/default/puppetdb+++</cite>+++ only has 192MB of heap space so we upped that to 1GB and problem solved! (at least for now)

+++<cite>+++/etc/default/puppetdb+++</cite>+++

[source,text]
----

###########################################
# Init settings for puppetdb
###########################################

# Location of your Java binary (version 6 or higher)
JAVA_BIN="/usr/bin/java"

# Modify this if you'd like to change the memory allocation, enable JMX, etc
JAVA_ARGS="-Xmx1024m"

# These normally shouldn't need to be edited if using OS packages
USER="puppetdb"
INSTALL_DIR="/usr/share/puppetdb"
CONFIG="/etc/puppetdb/conf.d"
----
The Curse Of Knowledge
Date: 2012-08-28 21:22:36

My colleague https://twitter.com/anand003/[Anand Vishwanath] recently recommended the book 'http://www.amazon.co.uk/Made-Stick-ideas-others-unstuck/dp/009950569X/ref=sr_1_1?ie=UTF8&qid=1346108348&sr=8-1[Made To Stick]' and one thing that has really stood out for me while reading it is the idea of the 'The Curse Of Knowledge' which is described like so:

____
Once we know something, we find it hard to imagine what it was like not to know it. Our knowledge has "cursed" us. And it becomes difficult for us to share out knowledge with others, because can't readily re-create our listeners' state of mind.
____

This is certainly something I imagine that most people have experienced, perhaps for the first time at school when we realised that the best teacher of a subject isn't necessarily the person who is best at the subject.

I'm currently working on an infrastructure team and each week every team does a mini showcase where they show the other teams some of the things they've been working on.

It's a very mixed audience - some very technical people and some not as technical people - so we've found it quite difficult to work out how exactly we can explain what we're doing in a way that people will be able to understand.

A lot of what we're doing is quite abstract/not very visible and the first time we presented we assumed that some things were 'obvious' and didn't need an explanation.

Having experienced a lot of blank looks we learnt that *they're only obvious if you spend the whole day doing them*.

We needed to find a way to make what we were doing a bit more accessible by closing the gap between what people currently knew and our topic.

One effective way of doing this is to step up a level of abstraction and describe what the goal is rather than spending too long on the implementation details.

An example of something we presented was http://logstash.net/[logstash], a tool used to collect logs from a bunch of different sources and then allows us to run regular expressions over them to make the data more easily searchable.

When explaining this in the demo a colleague explained its use from the point of view of something going wrong with one of the applications and the need to fix it quickly.

He pointed out that it's much easier to do this if all the information we need is in one place and is easy to search, thereby explaining why logstash was useful.

By weaving our use of logstash into a more general story that people do understand we've been able to make this and other technical things that we're working on more accessible to others.

We also realised that people tend to forget what we've talked about previously so now when we present something we spend a bit of time refreshing people's minds so that they have a reference point to work from.

All the things we've tried try to make what can see abstract more concrete which is one of the six ways that the authors of 'Made To Stick' suggest to make our ideas more memorable.

It's a good book, worth flicking through at the very least!
SSHing onto machines via a jumpbox
Date: 2012-08-10 00:58:46

We wanted to be able to ssh into some machines which were behind a firewall so we set up a http://blog.industrialdefender.com/?p=612[jumpbox]  which our firewall directed any traffic on port 22 towards.

Initially if we wanted to SSH onto a machine inside the network we'd have to do a two step process:

[source,text]
----

$ ssh jumpbox
# now on the jumpbx
$ ssh internal-network-machine
----

That got a bit annoying after a while so http://www.linkedin.com/in/samsharpe[Sam] showed us a neat way of proxying the second ssh command through the first one by making use of http://netcat.sourceforge.net/[netcat].

We put the following into +++<cite>+++~/.ssh/config+++</cite>+++:

[source,text]
----

Host jumpbox jumpbox-ip
 Hostname jumpbox-ip
 User     user
 IdentityFile ~/.ssh/id_rsa
 ProxyCommand none

Host internal-network-machine
  Hostname internal-network-machine-ip

Host 10.*
 User     ubuntu
 ProxyCommand ssh jumpbox exec nc -w 9000 %h %p
 UserKnownHostsFile /dev/null
 StrictHostKeyChecking no
----

The '-w 9000' flag defines a 2 1/2 hour wait period so that any http://www.lofar.org/wiki/doku.php?id=public:ssh-usage[orphaned connections will die off] within that time.

%h and %p represent the host and port of the internal machine so in this case %h is 'internal-network-machine-ip' and the port will be 22.

We can then just do the following to ssh into the machine:

[source,text]
----

ssh internal-network-machine
----

Which is pretty neat!

This is http://benno.id.au/blog/2006/06/08/ssh_proxy_command[explained further on benno's blog] and on the http://www.undeadly.org/cgi?action=article&sid=20070925181947[Open BSD journal].
VCloud Guest Customization Script : [: postcustomization: unexpected operator
Date: 2012-08-06 21:50:07

We have been doing some work to automatically provision machines using the VCloud API via fog and one of the things we wanted to do was http://blogs.vmware.com/vsphere/2012/06/using-a-guest-customization-script-to-tell-when-vappvm-is-ready-in-vcloud-director.html[run a custom script the first time that a node powers on].

The following http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026614[explains how customization scripts work]:

____
In vCloud Director, when setting a customization script in a virtual machine, the script:

* Is called only on initial customization and force recustomization.
* Is called with the precustomization command line parameter before out-of-box customization begins.
* Is called with the postcustomization command line parameter after out-of-box customization finishes.
* Needs to be a batch file for Windows virtual machines and a shell script for Unix virtual machines.
____

We wanted the script to run only when passed the 'postcustomization' flag because our script relied on some networking configuration which hadn't yet been done in the 'precustomization' state.

We wrote something like the following script:

[source,text]
----

#!/bin/bash
if [ x$1 == x"postcustomization" ]; then
  echo post customization
fi
----

Unfortunately when we provisioned the node it hadn't run any of the code within the if block and we saw the following message in +++<cite>+++/var/log/vmware-inc/customization.log+++</cite>+++:

[source,text]
----

5: [: xpostcustomization: unexpected operator
----

https://twitter.com/nickstenning[Nick] pointed out that the http://pubs.opengroup.org/onlinepubs/9699919799/utilities/test.html#top[test] utility which we're using to do the comparison on the 2nd line uses a single = in the POSIX shell even though it will work with double = in the bash shell.

We thought this was pretty strange since we are telling the script to run with the bash shell in the first line.

We eventually realised that the script was being spawned out to a POSIX shell by +++<cite>+++/root/.customization/post-customize-guest.sh+++</cite>+++ which is the script that gets called on power on:

[source,text]
----

 ((${SH} $POST_CUSTOMIZATION_TMP_SCRIPT_NAME "postcustomization" > /tmp/stdout.log) 2>&1 | ${TEE} -a /tmp/stderr.log)
----

I created a simple script to check the theory:

[source,text]
----

#!/bin/bash

[ "mark" == "mark" ] && echo Mark
----

Which works fine when called directly:

[source,text]
----

$ ./mark.sh
Mark
----

And throws the expected error when called with 'sh':

[source,text]
----

$ sh mark.sh
[: 3: mark: unexpected operator
----

We therefore needed to change our script to do the comparison with a single = like so:

[source,text]
----

#!/bin/bash
if [ x$1 = x"postcustomization" ]; then
  echo post customization
fi
----
Installing Puppet on Oracle Linux
Date: 2012-01-18 00:30:59

We've been spending some time trying to setup our developer environment on a Oracle Linux 5.7 build and one of the first steps was to install Puppet as we've already created scripts which automate the installation of most things.

Unfortunately Oracle Linux builds don't come with any yum repos configured so when you run the following command...

[source,text]
----

ls -alh /etc/yum.repos.d/
----

...you don't see anything :(

We eventually realised that there are a http://public-yum.oracle.com/[list of public yum repositories on the Oracle website], of which we needed to download the definition for Oracle Linux 5 like so:

[source,text]
----

cd /etc/yum.repos.d
wget http://public-yum.oracle.com/public-yum-el5.repo
----

We then need to edit that file to enable the appropriate repository. In this case we want to enable +++<cite>+++ol5_u7_base+++</cite>+++:

[source,text]
----

[ol5_u7_base]
name=Oracle Linux $releasever - U7 - $basearch - base
baseurl=http://public-yum.oracle.com/repo/OracleLinux/OL5/7/base/$basearch/
gpgkey=http://public-yum.oracle.com/RPM-GPG-KEY-oracle-el5
gpgcheck=1
enabled=1
----

I made the mistake of enabling +++<cite>+++ol5_u5_base+++</cite>+++ which led to us getting some really weird problems whereby yum got confused as to which version of +++<cite>+++libselinux+++</cite>+++ we had installed and was therefore unable to install +++<cite>+++libselinux-ruby+++</cite>+++ as its dependencies weren't being properly satisfied.

Calling 'yum list installed' suggested that we had +++<cite>+++libselinux+++</cite>+++ 1.33.4.5-7 installed but if we ran 'yum install libselinux' then it suggested we already had 1.33.4.5-5 installed. Very confusing!

After trying to uninstall and downgrade +++<cite>+++libselinux+++</cite>+++ and pretty much destroying the installation in the process, another colleague spotted my mistake.

We also found that we had to add the epel repo which gave us access to some other packages that we needed:

[source,text]
----

rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
----

After all that was done we were able to run the command to install puppet:

[source,text]
----

yum install puppet
----

That installs puppet 2.6.12 as that's the latest version in that repo. The latest stable version is 2.7.9 but I think we'll need to hook up a puppet specific repo to get that working.
Application footprint
Date: 2012-01-16 01:40:32

I recently came across Carl Erickson's 'http://spin.atomicobject.com/2012/01/11/small-teams-are-dramatically-more-efficient-than-large-teams/[small teams are dramatically more efficient than large teams]' blog post which reminded me of something which my colleague https://twitter.com/#!/a5hok[Ashok] suggested as a useful way for determining team size - *the application footprint*.

As I understand it the application footprint is applicable for an application at a given point in time and determines *how many parallel tasks/streams of work we have*.

In the case of the project that I'm currently working on there are 3 separate components which need to interact with each other via an API but otherwise are independent.

image::{{<siteurl>}}/uploads/2012/01/footprint.jpg[Footprint,240]

We can therefore have 3 pairs working - one on each component - and won't have to worry about them stepping on each other's toes.

One interesting thing about the application footprint is that *it doesn't stay the same size all the time*.

More often than not once a team has gained  trust by getting a release out the product owner will start prioritising more independent features which don't necessarily overlap.

At this stage it might not be such a bad idea to add people to the team if we want to try and finish more quickly.

If we're already at the point where we have the same number of pairs as parallel pieces of work then adding people is going to be problematic because we'll struggle to find work for everyone to do.

Stories in the same stream will have dependencies on each other and although it's theoretically possible to start on something which has a dependency, the likelihood of having to rework it is higher.

One way to get around that problem if we decide that we don't want to reduce our team size is to have a pair assigned to working on bugs, cross functional requirements such as performance testing/tuning or doing some technical analysis on upcoming stories.

It's easy enough to remember all this when you're starting out building an application but I think it's something that we need to keep in mind so that if there's pressure to add people to 'go faster' then we can determine if that will actually be the case.

_As an aside_

Obviously there are times when we decide that we're happy to put more people on a team than it's footprint might suggest in order to get an overall gain.

For example with 5 pairs we may finish 50 points in a week but if we increase to 10 pairs then perhaps we now get 60 points.

We've nearly halved the efficiency of each pair but overall we've got a marginal gain which sometimes makes sense. We also need to be aware of the http://www.markhneedham.com/blog/2011/02/16/increasing-team-sizes-collective-unresponsibility/[collective unresponsibility] that we might introduce by doing this.

_Photo http://www.flickr.com/photos/farlane/2541022957/sizes/s/in/photostream/[courtesy of farlane]_
Focused Retrospectives: things to watch for
Date: 2012-01-16 01:01:30

A few weeks ago a slide deck from an http://www.slideshare.net/estherderby/agile-retrospectives-4976896[Esther Derby presentation on retrospectives] was doing the rounds on twitter and one thing that I found interesting in the deck was the suggestion that a retrospective needs to be focused in some way.

I've participated in a few focused retrospectives over the past 7/8 months and I think there are some things to be careful about when we decide to focus on something specific rather than just looking back at a time period in general.

== Victimisation

In a retrospective about 6 months ago or so we focused on the analysis part of our process as we'd been struggling to know when a story was complete and what exactly its scope was.

The intention wasn't the victimise the people working in that role but since there were very few of them compared to people in other roles they were forced onto the defensive as people criticised their work.

It was a very awkward retrospective and it felt like a retrospective was probably the wrong place to address the problem.

It might have been better for the analysts to have been given the feedback privately and then perhaps worked on a solution with a smaller group of people.

== Looking for a problem when there isn't one

I had an interesting conversation with a colleague about whether with very focused retrospectives we end up looking for something to change rather than having any specific pain point which necessitates change.

The problem with this is that there's a thin line between following the status quo because it works and getting complacent and not looking for ways to improve.

It is interesting to keep in mind though that *if it doesn't seem like there is something to change in an area then perhaps that's the wrong thing to be focusing on at the moment*, which nicely leads into...

== Let the team choose the area of focus

There can be a tendency in the teams I've worked on for people in managementy roles to dictate what the focus of the retrospective will be which makes sense in a way since they may be able to see something which the team can't.

On the other hand it can mean that we end up focusing on the wrong thing and team members probably won't be that engaged in the retrospective since they don't really get to dictate what's talked about.

Esther points this out out on slide 23 of the presentation - "*Choose a focus that reflects what's going on for the team*". This perhaps can be determined by having a vote before hand based on some topics that seem prominent.

== In summary

There's lots of other useful tips in Esther's slide deck which are worth having a look at and I'm sure most of the potential problems I've listed probably don't happen when we have a highly skilled/experienced facilitator.
Oracle: dbstart - ORACLE_HOME_LISTNER is not SET, unable to auto-start Oracle Net Listener
Date: 2012-01-26 21:58:27

We ran into an interesting problem when trying to start up an Oracle instance using +++<cite>+++dbstart+++</cite>+++ whereby we were getting the following error:

[source,text]
----

-bash-3.2$ dbstart

ORACLE_HOME_LISTNER is not SET, unable to auto-start Oracle Net Listener
Usage: /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbstart ORACLE_HOME
Processing Database instance "orcl": log file /u01/app/oracle/product/11.2.0/dbhome_1/startup.log
----

Ignoring the usage message we thought that setting the environment variable was what we needed to do, but...

[source,text]
----

-bash-3.2$ export ORACLE_HOME_LISTNER=$ORACLE_HOME
-bash-3.2$ dbstart
ORACLE_HOME_LISTNER is not SET, unable to auto-start Oracle Net Listener
Usage: /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbstart ORACLE_HOME
Processing Database instance "orcl": log file /u01/app/oracle/product/11.2.0/dbhome_1/startup.log
----

We ended up looking at the source of +++<cite>+++dbstart+++</cite>+++ to see what was going on:

[source,text]
----

# First argument is used to bring up Oracle Net Listener
ORACLE_HOME_LISTNER=$1
if [ ! $ORACLE_HOME_LISTNER ] ; then
  echo "ORACLE_HOME_LISTNER is not SET, unable to auto-start Oracle Net Listener"
  echo "Usage: $0 ORACLE_HOME"
----

The usage message does explain that you're supposed to call it like this:

[source,text]
----

-bash-3.2$ dbstart $ORACLE_HOME
----

But it still seems a bit weird/misleading to me that you'd override the value of a global variable from inside a script which doesn't suggest that it's going to do that!

Such is life in Oracle land..