markhneedham/blog huggingface dataset:
Nygard Big Data Model: The Investigation Stage
Date: 2012-10-10 00:00:36

Earlier this year http://thinkrelevance.com/blog/2012/04/04/big-data-reference-model[Michael Nygard wrote an extremely detailed post about his experiences in the world of big data projects] and included in the post was the following diagram which I've found very useful.

image::{{<siteurl>}}/uploads/2012/10/nygard.png[Nygard,600]

_Nygard's Big Data Model (shamelessly borrowed by me because it's awesome)_

https://twitter.com/a5hok[Ashok] and I have been doing some work in this area helping one of our clients make sense of and visualise some of their data and we realised retrospectively that *we were very acting very much in the investigation stage of the model*.

In particular Nygard makes the following suggestions about the way that we work when we're in this mode:

____
We *don't want to invest in fully automated machine learning and feedback*. That will follow once we validate a hypothesis and want to integrate it into our routine operation. Ad-hoc analysis refers to human-based data exploration. This can be as simple as spreadsheets with line graphs\...the key aspect is that most of the tools are interactive. Questions are expressed as code, but that *code is usually just "one shot"* and is not meant for production operations.
____

In our case we weren't doing anything as complicated as machine learning - most of our work was working out the relationships between things, how best to model those and what visualisations would best describe what we were seeing.

We didn't TDD any of the code, we copy/pasted a lot and when we had a longer running query we didn't try and optimise it there and then, instead we ran it once, saved the results to a file and then used the file to load it onto the UI.

We were able to work in iterations of 2/3 hours during which we tried to http://www.markhneedham.com/blog/2012/05/05/neo4j-what-question-do-you-want-to-answer/[answer a question] (or more than one if we had time) and then showed our client what we'd managed to do and then decided where we wanted to go next.

To start with we did all this with a subset of the actual data set and then once we were on the right track we loaded in the rest of the data.

We can easily get *distracted by the difficulties of loading large amounts of data before checking whether what we're doing makes sense*.

We iterated 4 or 5 different ideas before we got to one that allowed us to explore an area which hadn't previously been explored.

Now that we've done that we're rewriting the application from scratch, still using the same ideas as from the initial prototype, but this time making sure the queries can run on the fly and making the code a bit closer to production quality!

We've *moved into the implementation stage of the model* for this avenue although if I understand the model correctly, it would be ok to go back into investigation mode if we want to do some discovery work with others parts of the data.

I'm probably quoting this model way too much to people that I talk to about this type of work but I think it's really good so nice work Mr Nygard!
How we're using story points
Date: 2012-10-21 23:08:01

A couple of weeks ago Joshua Kerievsky wrote a post http://www.industriallogic.com/blog/stop-using-story-points/[describing how he and his teams don't use story points anymore] because of the problems they'd had with them which included:

* Story Point Inflation - inflating estimates of stories so that the velocity for an iteration is higher
* Comparing teams by points - judging comparative performance of teams by how many points they're able to complete

On the team I'm currently working on we still estimate the relative size of stories using points but we don't use velocity per iteration to keep score - most of the time it's barely even mentioned.

Instead for the past couple of months we've just been *using the velocity to see whether or not we were going to achieve the minimum viable infrastructure (MVI)* that we needed to have done before launch.

The nice thing on this occasion was that the MVI was actually reasonably flexible and there were bits of it which were 'nice to have' if we had the time and would make our life easier but didn't have to be there to launch.

As it turned out we ended up finishing the cards required for the MVI a couple of weeks early which left us some flexibility/time to do things which had been forgotten or cropped up late on because something else didn't work.

We did have a few weeks where our velocity fluctuated massively but an interesting observation which https://twitter.com/philandstuff[Phil] made at the time was that despite the points totals being different we'd actually completed roughly the same number of cards.

Joshua describes the same thing in his post when detailing an email sent to the Extreme Programming list by Don Wells:

____
We have been counting items done. Each week we just choose the most important items and sign up for them up to the number from last week. It turns out that we get about the same number of them done regardless of estimated effort
____

This approach was described to me a few years ago by http://blog.rufiao.com/about/[Julio Maia] but as I understand it works on the assumption that we can break the work down into chunks which are roughly the same size which in my experience is very difficult to do.

Whatever approach we end up using I think it's important that we *don't praise or criticise the team based on the velocity achieved that week*.

I've worked on numerous teams where the project manager will praise the team in the standup for achieving a velocity higher than normal even though nothing has changed which would account for the increase.

In fact the change in velocity can probably be accounted for by normal variance.

If we don't do that and just use the story points as a tool for roughly judging how much work we can complete in a time period then they're not so horrendous.

Easier said than done of course!
Do the simple thing
Date: 2012-10-21 21:35:35

One of the most unexpected things that I picked up while pairing with https://twitter.com/a5hok[Ashok] for a few days in August/September is his ability to *pick the simplest solution* when confronted with a problem.

On numerous occasions we'd be trying to do something and I'd end up on a yak shaving mission trying to get a complicated approach to work while he watched on with bemusement.

I thought I'd actually learnt this lesson from working with Ashok but on a couple of occasions over the last week I've caught myself doing the same thing again!

A couple of days ago I wanted to send a photo taken on my phone to someone but the photo was about 2MB in size, I couldn't get the wireless network in the office to work and attempting to send it via 3G was totally failing.

I somehow found myself trying to debug the problem with the wireless network when http://www.linkedin.com/pub/shodhan-sheth/2/277/287[Shodhan] pointed out that I could just transfer the file via USB to my machine and then email it from there.

To add to my shame, earlier this evening I was trying to help my mum get a song from a CD onto the hard drive and somehow found myself trying to compile http://ffmpeg.org/download.html[ffmeg] so that I could convert an AIFF file into an MP3 on my Mac.

In actual fact all I needed to do was rip the CD using Windows Media Player which somehow had completely skipped by mind until https://twitter.com/jennifersmithco[Jen] and http://www.youtube.com/watch?v=rV2wkX0InBk[YouTube] led me to that solution at the same time.

In retrospect I can't believe I actually believed that what I was doing was a good solution to the problem but some part of my brain clearly though it was!

Although both this examples are on the periphery of software development I think it's very easy to fall into the same trap there - *we want to use things that we've learned to come up with clever solutions* but that cleverness isn't always useful!
Mac OS X: Removing Byte Order Mark with an editor
Date: 2012-10-07 10:43:46

About a month ago I wrote about http://www.markhneedham.com/blog/2012/09/03/a-rogue-357273277-utf-8-byte-order-mark/[some problems I was having working with Windows generated CSV files] which had a Byte Order Mark (BOM) at the beginning of the file and I described a way to get rid of it using awk.

It's a bit of a long winded process though and I always forget what the parameters I need to pass to awk are so I thought it would probably be quicker if I could just work out a way to get rid of the BOM using an editor.

I'm using a Mac so the most popular hex editor on that platform seems to be http://ridiculousfish.com/hexfiend/[HexFiend].

If we open the problematic file with that it's reasonably easy to see where the BOM is and we can then manually remove it.

image::{{<siteurl>}}/uploads/2012/10/bom.jpg[Bom,600]

There is a http://stackoverflow.com/questions/2160405/hex-editor-for-mac-osx[list of other hex editors for the Mac on this Stack Overflow thread].

I figured there was probably a way to do this using emacs and indeed there is!

One way is to open the file using 'Meta-X find-file-literally' which displays all non ASCII characters so that you can delete them if you want:

image::{{<siteurl>}}/uploads/2012/10/bom-2.jpg[Bom 2,600]

I also http://stackoverflow.com/questions/3859274/removing-byte-order-marks-boms-using-emacs[learnt about another way] which is to first open the file using 'Ctrl X Ctrl F' and then run 'Meta-X set-buffer-file-encoding-system' and enter 'utf-8' before saving the file. The BOM will now be deleted!

This is perhaps a bit simpler since you don't need to delete the characters manually.

There is a http://unix.stackexchange.com/questions/39786/in-emacs-or-other-editor-how-to-display-the-byte-offset-of-the-cursor[third way where you open the file using 'Meta X hexl-find-file'] but it seems more difficult to use than the other two options!
Play Framework 2.0: Rendering JSON data in the view
Date: 2012-10-14 09:28:28

I've been playing around with the http://www.playframework.org/[Play Framework] which we're using to front a bunch of visualisations and one thing I wanted to do is send a data structure to a view and then convert that into JSON.

I've got a simple controller which looks like this:

[source,java]
----

package controllers;

import play.mvc.Controller;
import play.mvc.Result;
import views.html.*;

public class SalesByCategory extends Controller {
    public static Result index() {
        ArrayList<Map<String, Object>> series = new ArrayList<Map<String, Object>>();
        Map<String, Object> oneSeries = new HashMap<String, Object>();
        oneSeries.put("name", "awesome");
        oneSeries.put("sales", calculateSales()); # would call a method elsewhere, implementation isn't important

        series.add(oneSeries);

        # I have a view named 'index.scala.html'
        return ok(index.render("Awesome visualisation", series));
    }
}
----

The top of the corresponding view looks like this:

[source,html]
----

@(message: String)(series:List[Map[String, Object]])
----

I'm using the GSON library to convert objects into JSON so I need to import that into the view:

[source,html]
----

@import com.google.gson.Gson
----

I was initially struggling to work out what the syntax would be to call the GSON code from within the page but with a bit of trial and error realised that the following would do the trick:

[source,html]
----

<script lang="text/javascript">
  var series = @{new Gson().toJson(series)};
</script>
----

The problem with this version of the code was that the string was being escaped so I ended up with series having a value like this:

[source,text]
----

[{&quot;name&quot;:&quot; #and so on!
----

I needed to tell Play not to escape this string which in Play v1 was done by calling 'raw()' on the string but http://stackoverflow.com/questions/10326050/getting-a-raw-string-back-for-use-in-javascript-in-play-framework-2-0[in Play v2 is done using the 'Html' method]:

[source,html]
----

<script lang="text/javascript">
  var series = @{Html(new Gson().toJson(series))};
</script>
----

And now the JSON renders beautifully!
Debugging: Google vs The Manual
Date: 2012-07-04 00:00:07

Over the last six months or so I've worked with a bunch of different people and one of the things that I've noticed is that when something isn't working there tend to be two quite distinct ways that people go about trying to solve the problem.

== The Manual

The http://en.wikipedia.org/wiki/RTFM[RTFM] crowd will go straight for the official documentation or source code if needs be in an attempt to work through the problem from first principals.

While this approach is more admirable it isn't always quicker, especially if the reason that something's gone wrong is because of an obscure dependency or something similar. That type of problem is unlikely to be documented!

On the other hand, if you're an early adopter for something then you probably have no choice other than to read the source since the documentation probably doesn't yet exist!

== Google

The lazier amongst us will copy the error message and keep deleting parts of it until Google turns up a useful result.

This approach works pretty well when working with popular software/tools where the likelihood of someone coming across the same problem and posting about it on a mailing list or blog is higher.

It's often possible to work backwards from a similar problem that someone else has to solve your own problem.

This approach works less well for obscure and/or proprietary software/tools where the community is less established and people haven't reported problems.

== Overall

A combination of the two approaches is what most people end up doing, although it's interesting watching how long they'll persist with their favoured approach before switching!

I fall in the second camp and a reasonable percentage of my blog posts consist of me recording problems I experienced so that other people don't have to search for ages if they come across the same thing in future.
Bash Shell: Reusing parts of previous commands
Date: 2012-07-05 23:42:35

I've paired a few times with my colleague https://twitter.com/#!/philandstuff[Phil Potter] over the last couple of weeks and since he's a bit of a ninja with bash shortcuts/commands I wanted to record some of the things he's shown me so I won't forget them!

Let's say we're in the '/tmp' directory and want to create a folder a few levels down but forget to pass the '-p' option to 'mkdir':

[source,text]
----

$ mkdir blah/de/blah
mkdir: cannot create directory `blah/de/blah': No such file or directory
----

One way of fixing that would be to press the up arrow and navigate along the previous command and put in the '-p' flag but it's a bit fiddly so instead we can do the following:

[source,text]
----

$ ^mkdir^mkdir -p
mkdir -p blah/de/blah
----

The '^' allows us to replace any parts of the previous command and then run it again, so we could actually make that more concise if we wanted to:

[source,text]
----

$ ^r^r -p
mkdir -p blah/de/blah
----

Reasonably frequently after we've created a folder like this we'll want to create a file inside it. '!$' comes in handy here as it allows us to refer to the last argument passed to the last command:

[source,text]
----

$ touch !$/blah.xml
touch blah/de/blah/blah.xml
----

If we decide to remove that file and want to check it's been deleted we can run the following:

[source,text]
----

$ touch blah/de/blah/blah.xml
$ rm blah/de/blah/blah.xml
$ ls -alh !$:h
ls -alh blah/de/blah
total 8.0K
drwxr-xr-x 2 mneedham mneedham 4.0K 2012-07-05 16:26 .
drwxr-xr-x 3 mneedham mneedham 4.0K 2012-07-05 16:16 ..
----

The ':h' http://www.gnu.org/software/bash/manual/html_node/Modifiers.html#Modifiers[modifier] removes the file name and leaves the rest of the file path alone.

We can expand the value of '!$' or any other command by typing 'Esc' followed by '^' (Shift 6):

[source,text]
----

mneedham@ubuntu:/tmp$ mkdir -p blah/de/blah
mneedham@ubuntu:/tmp$ touch !$
----

Esc + {caret}

[source,text]
----

$ mkdir -p blah/de/blah
$ touch blah/de/blah
----

If we get carried away with modifiers we could also fix that first 'mkdir' command by making use of the 'substitution' modifier:

[source,text]
----

$ mkdir blah/de/blah
mkdir: cannot create directory `blah/de/blah': No such file or directory

$ !!:s/r/r -p
mkdir -p blah/de/blah
----

There's not really any reason I can think of why you'd want to use that when you can use the initial '^' approach though!

I came across http://samrowe.com/wordpress/advancing-in-the-bash-shell/[this blog post which explains how to do this type of thing and much more in the bash shell] - worth a read!
Racket: Wiring it up to a REPL ala SLIME/Swank
Date: 2012-07-11 19:34:34

One of the awesome things about working with clojure is that it's possible to wire up clojure files in emacs to a REPL by making use of http://www.beaconhill.com/solutions/kb/clojure/setting-up-clojure-with-slime-on-ubuntu.html[Slime]/https://github.com/technomancy/swank-clojure[Swank].

I've started using http://racket-lang.org/[Racket] to work through the examples in http://www.amazon.co.uk/The-Little-Schemer-Daniel-Friedman/dp/0262560992/ref=sr_1_1?ie=UTF8&qid=1342034533&sr=8-1[The Little Schemer] and wanted to achieve a similar thing there.

____
Racket is a modern programming language in the Lisp/Scheme family, suitable for a wide range of applications
____

I don't know much about configuring emacs so I made use of Phil Halgelberg's https://github.com/technomancy/emacs-starter-kit[emacs-starter-kit which is available on github].

On my travels I came across http://www.youtube.com/watch?v=Zev99LT-k70[this video describing how to do exactly what I wanted].

The first step was to download a library called 'http://www.nongnu.org/geiser/[geiser]':

____
Geiser is a generic Emacs/Scheme interaction mode, featuring an enhanced REPL and a set of minor modes improving Emacs' basic scheme major mode.
____

The easiest way seemed to be to clone the git repository:

[source,text]
----

git clone git://gitorious.org/geiser/mainline.git geiser
----

I then added the following line to my +++<cite>+++/Users/mneedham.emacs.d/init.el+++</cite>+++ file:

[source,text]
----

(load "/Users/mneedham/github/geiser/elisp/geiser.el")
----

From emacs you then need to run the following command:

[source,text]
----

M-x geiser
----

It then asks you which scheme implementation you want to use, so in our case we need to make sure that racket is available on the path.

For some reason when I downloaded racket it seemed to prefer mounting itself as a volume rather than installing itself to /Applications so I ended up copy/pasting the folder to my home directory and adding it to the PATH in my +++<cite>+++.bashrc+++</cite>+++:

[source,text]
----

export PATH=$PATH:"/Users/mneedham/Documents/Racket v5.2.1/bin"
----

Now we can just type 'racket' when we're asked which scheme implementation we want and then we're done!

image::{{<siteurl>}}/uploads/2012/07/emacs-geiser1.jpg[Emacs geiser,600]
Puppet: Keeping the discipline
Date: 2012-07-29 21:53:03

For the last 5 weeks or so I've been working with http://puppetlabs.com/[puppet] every day to automate the configuration of various nodes in our stack and my most interesting observation so far is that you really need to keep your discipline when doing this type of work.

We can keep that discipline in three main ways when developing modules.

== Running from scratch

Configuring various bits of software seems to follow the http://en.wikipedia.org/wiki/Pareto_principle[80/20 rule] and we get very close to having each thing working quite quickly but then end up spending a disproportionate amount of time tweaking the last little bits.

When it's 'just little bits' there is a massive temptation to *make changes manually and then retrospectively put it into a puppet module and assume that it will work* if we run from scratch.

From doing that a few times and seeing a module fail when run at a later stage we've realised it isn't a very good approach and we're started to get into the discipline of starting from scratch each time.

Ideally that would mean creating snapshots of a Virtual Machine instance and rolling back after each run but that ends up taking quite a bit of time so more often we're removing everything that a module installs and then running that module again.

== Running as part of environment

We tend to run the modules individually when we start writing them but it's also important to run them as part of an environment definition as well to make sure modules play nicely with each other.

This is particularly useful if there are dependencies between our module and another one - perhaps our module will be notified if there's a configuration file change in another module for example.

== Running on different machines

Even if we do the above two things we've still found that we catch problems with our modules when we try them out on other machines.

Machines which have been in use for a longer period of time will have earlier versions of a module which aren't easy to upgrade and we'll need to remove the old version and install the new one.

We've also come across occasions where other machines have left over files from modules that have since been removed and although the files seem harmless they can interfere with new modules that we're writing.
Unix: tee
Date: 2012-07-29 19:11:24

I've read about the Unix 'http://en.wikipedia.org/wiki/Tee_(command)[tee]' command before but never found a reason to use it until the last few weeks.

One of the things I repeatedly do by mistake is open +++<cite>+++/etc/hosts+++</cite>+++ without sudo and then try to make changes to it:

[source,text]
----

$ vi /etc/hosts
# Editing it leads to the dreaded 'W10: Changing a readonly file'
----

I always used to close the file and then re-open it with sudo but I recently came across http://www.commandlinefu.com/commands/view/1204/save-a-file-you-edited-in-vim-without-the-needed-permissions[an approach which allows us to use 'tee' to get around the problem].

Once we've made our change we can run the following command:

[source,text]
----

:w !sudo tee %
----

The ':w' is usually used to write the buffer to the current file but here we're substituting a shell command to receive the buffer instead.

In this case we're sending the buffer to the 'sudo tee %' command where '%' refers to the current file name.

tee is defined like so:

____
The tee utility copies standard input to standard output, making a copy in zero or more files. The output is unbuffered.
____

So we're sending the buffer for the current file to tee which then saves it to the current file. Since we've run tee with sudo it will give us the appropriate permissions to write to the file that we didn't initially have.

This http://stackoverflow.com/questions/2600783/how-does-the-vim-write-with-sudo-trick-work[Stack Overflow post explains what's going on in more detail].

Another thing we've been working on is hooking up the logs of our various services to http://logstash.net/[logstash] and to test it out we needed to write to files in the +++<cite>+++/var/log+++</cite>+++ directory.

We started out trying to echo a message to the end of the apache log file like this:

[source,text]
----

$ echo "random message" >> /var/log/apache2/access_log
zsh: permission denied: /var/log/apache2/access_log
----

As you can see that doesn't work since we don't have permissions to write to files in +++<cite>+++/var/log+++</cite>+++ so we need to find another way to do that and https://twitter.com/nickstenning[Nick] pointed out that tee would do the job here as well.

[source,text]
----

$ echo "random message" | sudo tee -a /var/log/apache2/access_log
random message
----

Passing the '-a' flag to tee means that it appends standard input to the file instead of replacing the whole file like before. If we don't want standard in to be directed to standard out we can redirect that to +++<cite>+++/dev/null+++</cite>+++ like so:

[source,text]
----

$ echo "random message" | sudo tee > /dev/null -a /var/log/apache2/access_log
----