markhneedham/blog huggingface dataset:
Micro Services: Plugging in 3rd party components
Date: 2012-12-04 23:38:39

Over the past few weeks I've been involved in conversations with different clients around micro services and one thing about this architecture that seems quite popular is the ability to easily plug in 3rd party components.

In one case we were talking through the design of a system which would calculate and then apply price optimisations on products. The parts of the system we were discussing looked roughly like this:

 image:{{<siteurl>}}/uploads/2012/12/micro-services.png[Micro services,357]

As per the annotations, one of the questions asked was whether it would be possible to start out with the assumption that each component would be custom built and then assess that decision after a few weeks.

An advantage of splitting each of the components into their own application is that we could reasonably easily plug in a 3rd party tool behind the boundary while keeping all the HTTP wiring as custom code.

This allows us to get going quickly and write simple stubs in place of the main logic of some of the components to begin with.

We can defer the integration/learning curve of the 3rd party component while we prove out the architecture as a whole. In addition we are not letting the 3rd party component drive the design of our system but instead allowing it to play a supporting role.

One final thing to note is that since each component is a separate application *it's much easier to have different teams working on each one* than it would be if they were all contained in the same application.

There would need to be communication between teams around the design of contracts between the services but after an initial period of churn hopefully those would become reasonably stable.
rsyncing to an AWS instance
Date: 2012-12-11 23:44:05

I wanted to try running some of the http://www.markhneedham.com/blog/category/machine-learning/[machine learning algorithms] that https://twitter.com/jennifersmithco[Jen] and I have been playing around with on a beefier machine so I thought spinning up an AWS instance would be the best way to do that.

I built the JAR with the appropriate algorithms on my machine and then wanted to copy it up onto an AWS instance.

I could have used +++<cite>+++scp+++</cite>+++ but I quite like the progress bar that you can get with +++<cite>+++rsync+++</cite>+++ and since the JAR had somehow drifted to a size of 47MB the progress bar was useful.

When I provisioned the machine I created a public/private key pair and I was able to ssh into the machine like this:

[source,text]
----

ssh -l ubuntu -i ~/Downloads/machinenursery.pem ec2-54-242-108-142.compute-1.amazonaws.com
----

I needed to tell rsync to use the pen file which I initially tried to do with the following command:

[source,text]
----

rsync --progress 'ssh -i /Users/markhneedham/Downloads/machinenursery.pem' -avz target/ ubuntu@ec2-54-242-108-142.compute-1.amazonaws.com:machinenursery
----

It seemed to ignore the pem file and I got a permission denied error when I ran this:

[source,text]
----

Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at /SourceCache/rsync/rsync-42/rsync/io.c(452) [sender=2.6.9]
----

Eventually http://alestic.com/2009/04/ubuntu-ec2-sudo-ssh-rsync[came across an article which explained a way around the problem] using RSH instead of SSH:

[source,text]
----

rsync --progress --rsh 'ssh -i /Users/markhneedham/Downloads/machinenursery.pem' -avz target/ ubuntu@ec2-54-242-108-142.compute-1.amazonaws.com:machine nursery
----

As I understand it RSH isn't secure but all I'm transferring is a JAR file so it didn't seem like too much of an issue.

I'm sure there must be a way to transfer this file using SSH but I've tried all the different flags and I can't figure it out so if you know how to please let me know!.
Haskell: Initialising a map
Date: 2012-12-29 19:27:46

I've been converting a variation of Kruskal's algorithm from https://github.com/mneedham/algorithms2/blob/master/clustering_big_graph_bitwise.rb[Ruby] into https://github.com/mneedham/algorithms2/blob/master/clustering.hs[Haskell] and one thing I needed to do was create a map of binary based values to node ids.

In Ruby I wrote the following code to do this:

[source,ruby]
----

nodes = [1,2,5,7,2,4]
@magical_hash = {}
nodes.each_with_index do |node, index|
  @magical_hash[node] ||= []
  @magical_hash[node] << index
end

=> {1=>[0], 2=>[1, 4], 5=>[2], 7=>[3], 4=>[5]}
----

From looking at the documentation it seemed like the easiest way to do this in Haskell would be to convert the nodes into an appropriate list and then call the +++<cite>+++http://www.haskell.org/ghc/docs/6.12.2/html/libraries/containers-0.3.0.0/Data-Map.html#v%3AfromList[fromList]+++</cite>+++ function to build the map.

I needed to get the data to look like this:

[source,haskell]
----

> let nodesMap = Data.Map.fromList [(1, [0]), (2, [1,4]), (5, [2]), (7, [3]), (4, [5])]
> Data.Map.assocs nodesMap
[(1,[0]),(2,[1,4]),(4,[5]),(5,[2]),(7,[3])]
----

The first step was to create a list of tuples with the nodes ids and values:

[source,haskell]
----

> zip [0..] [1,2,5,7,2,4]
[(0,1),(1,2),(2,5),(3,7),(4,2),(5,4)]
----

I wanted group the collection by value so that in this instance I could have the 2 nodes with a value of '2' mapping from the same key in the map.

The following code helped do that:

[source,haskell]
----

groupIgnoringIndex = groupBy (\(_,x) (_,y) -> x == y)
sortIgnoringIndex = sortBy (\(_,x) (_,y) -> x `compare` y)
----

[source,haskell]
----

> (groupIgnoringIndex . sortIgnoringIndex) (zip [0..] [1,2,5,7,2,4])
[[(0,1)],[(1,2),(4,2)],[(5,4)],[(2,5)],[(3,7)]]
----

I wrote the following function to convert that collection into one that could be passed to +++<cite>+++fromList+++</cite>+++:

[source,haskell]
----

asMapEntry :: [(Int, Int)] -> (Int, [Int])
asMapEntry nodesWithIndexes = ((snd . head) nodesWithIndexes,
                              foldl (\acc (x,_) -> acc ++ [x]) [] nodesWithIndexes)
----

[source,haskell]
----

> asMapEntry [(1,2),(4,2)]
(2,[1,4])
----

We can then put all those functions together into the following top level function:

[source,haskell]
----

toMap :: [Int] -> Map Int [Int]
toMap nodes = Data.Map.fromList $ map asMapEntry $ (groupIgnoringIndex . sortIgnoringIndex) nodesWithIndexes
              where nodesWithIndexes = (zip [0..] nodes)
----

[source,haskell]
----

> Data.Map.assocs $ toMap [1,2,5,7,2,4]
[(1,[0]),(2,[4,1]),(4,[5]),(5,[2]),(7,[3])]
----

I haven't properly investigated all the functions available in the +++<cite>+++Data.Map+++</cite>+++ package but my gut feeling is there must be a better way to do this - the sort/group combination is ugly in the extreme!
Haskell: A cleaner way of initialising a map
Date: 2012-12-29 20:14:12

I recently wrote a blog post http://www.markhneedham.com/blog/2012/12/29/haskell-initialising-a-map/[showing a way of initialising a Haskell map] and towards the end of the post I realised how convoluted my approach was and wondered if there was an easier way and indeed there is!

To recap, this is the code I ended up with to populate a map with binary based values as the keys and node ids as the values:

[source,haskell]
----

import Data.Map

toMap :: [Int] -> Map Int [Int]
toMap nodes = fromList $ map asMapEntry $ (groupIgnoringIndex . sortIgnoringIndex) nodesWithIndexes
              where nodesWithIndexes = (zip [0..] nodes)

groupIgnoringIndex = groupBy (\(_,x) (_,y) -> x == y)
sortIgnoringIndex = sortBy (\(_,x) (_,y) -> x `compare` y)

asMapEntry :: [(Int, Int)] -> (Int, [Int])
asMapEntry nodesWithIndexes =
   ((snd . head) nodesWithIndexes, Prelude.foldl (\acc (x,_) -> acc ++ [x]) [] nodesWithIndexes)
----

[source,haskell]
----

> assocs $ toMap [1,2,5,7,2,4]
[(1,[0]),(2,[4,1]),(4,[5]),(5,[2]),(7,[3])]
----

To sum up what we're trying to do: when a key doesn't have an entry we want to create one with a list containing the appropriate value and if the key already has a value then we want to append that value to the existing list.

As it turns out the +++<cite>+++http://www.haskell.org/ghc/docs/6.12.2/html/libraries/containers-0.3.0.0/Data-Map.html#v%3AinsertWith[insertWith]+++</cite>+++ function does exactly what we want:

[source,haskell]
----

> let emptyMap = empty :: Map Int [Int]
> assocs $ foldl (\acc (id,val) -> insertWith (++) val [id] acc) emptyMap nodesWithIndexes
[(1,[0]),(2,[4,1]),(4,[5]),(5,[2]),(7,[3])]
----

Based on this experience it would appear that the same type of thing applies when coding in Haskell as when coding in Clojure.

To http://blog.jayfields.com/2012/09/replacing-common-code-with-clojureset.html[paraphrase Jay Fields]:

____
If you've written a fair amount of Clojure code [\...] then chances are you've probably reinvented a few functions that are already available in the standard library.
____
Sed: Replacing characters with a new line
Date: 2012-12-29 17:49:46

I've been playing around with writing some algorithms in both Ruby and Haskell and the latter wasn't giving the correct result so I wanted to output an intermediate state of the two programs and compare them.

I didn't do any fancy formatting of the output from either program so I had the raw data structures in text files which I needed to transform so that they were comparable.

The main thing I wanted to do was get each of the elements of the collection onto their own line. The output of one of the programs looked like this:

[source,text]
----

[(1,2), (3,4)…]
----

To get each of the elements onto a new line my first step was to replace every occurrence of ', (' with '\n('. I initially tried using +++<cite>+++sed+++</cite>+++ to do that:

[source,text]
----

sed -E -e 's/, \(/\\n(/g' ruby_union.txt
----

All that did was insert the string value '\n' rather than the new line character.

I've come across similar problems before and I usually just use +++<cite>+++tr+++</cite>+++ but in this case it doesn't work very well because we're replacing more than just a single character.

I came across http://www.linuxquestions.org/questions/linux-software-2/sed-insert-a-newline-why-does-not-it-work-158806/[this thread] on Linux Questions which gives a couple of ways that we can get see to do what we want.

The first suggestion is that we should use a back slash followed by the enter key while writing our sed expression where we want the new line to be and then continue writing the rest of the expression.

We therefore end up with the following:

[source,text]
----

sed -E -e "s/,\(/\
/g" ruby_union.txt
----

This approach works but it's a bit annoying as you need to delete the rest of the expression so that the enter works correctly.

An alternative is to make use of +++<cite>+++echo+++</cite>+++ with the '-e' flag which allows us to output a new line. Usually http://linux.about.com/library/cmd/blcmdl1_echo.htm[backslashed characters aren't interpreted] and so you end up with a literal representation. e.g.

[source,text]
----

$ echo "mark\r\nneedham"
mark\r\nneedham

$ echo -e "mark\r\nneedham"
mark
needham
----

We therefore end up with this:~~~ ~~~text sed -E -e "s/, \(/\\`echo -e '\n\r'`/g" ruby_union.txt ~~~

** Update **

It was pointed out in the comments that this final version of the sed statement doesn't actually lead to a very nice output which is because I left out the other commands I passed to it which get rid of extra brackets.

The following gives a cleaner output: ~~~text $ echo "[(1,2), (3,4), (5,6)]" | sed -E -e "s/, \(/\\`echo -e '\n\r'`/g" -e 's/\[|]|\)|\(//g' 1,2 3,4 5,6 ~~~
apt-get update: 416 Requested Range Not Satisfiable
Date: 2012-12-10 00:39:34

We were trying to run a puppet update on some machines last week and one of the first things it does is run 'apt-get update' which was working on all but one node for which it was returning the following exception:

[source,text]
----

Err http://us-west-1.ec2.archive.ubuntu.com/ubuntu/ i386 Packages
  416 Requested Range Not Satisfiable
Fetched 5,079B in 2s (2,296B/s)
W: Failed to fetch http://us-west-1.ec2.archive.ubuntu.com/ubuntu/dists/maverick-updates/main/binary-i386/Packages.gz
416 Requested Range Not Satisfiable
----

It turns out one way that exception can manifest is if you've got a partial copy of the index files from the repository and in this case the solution was https://bugs.launchpad.net/ubuntu/+source/apt/+bug/798023[as simple as deleting those and trying again]:

[source,text]
----

sudo rm -rf /var/lib/apt/lists/partial/*
----

Running 'apt-get update' again after that worked perfectly and the world was a happy place again.
Haskell: Downloading the core library source code
Date: 2012-12-31 22:39:15

I've started playing around with Haskell again and since I'm doing so on a new machine I don't have a copy of the language source code.

I wanted to rectify that situation but my Google fu was weak and it took me way too long to figure out how to get it so I thought I'd better document it for future me.

The easiest way is to clone the copy of the GHC repository on github: ~~~text git clone https://github.com/ghc/ghc.git ~~~

Initially that doesn't have any of the code for the core libraries but running the following command (which takes ages!) sorts it out: ~~~text cd ghc ./sync-all get ~~~

Noticing that the core libraries weren't there initially I thought I must have done something wrong so I went to the documentation for the function that I wanted to see the source for - +++<cite>+++http://hackage.haskell.org/packages/archive/array/0.2.0.0/doc/html/Data-Array-MArray.html#v%3AgetAssocs[getAssocs]+++</cite>+++.

From that page there is http://hackage.haskell.org/packages/archive/array/0.2.0.0/doc/html/src/Data-Array-Base.html#getAssocs[a link to the source for that function] and http://hackage.haskell.org/packages/archive/array/0.2.0.0/doc/html/[a bit tweaking of the URL] lets us know that this function is defined in the +++<cite>+++array+++</cite>+++ package.

Most of the base packages are available from the http://darcs.haskell.org/packages/[Haskell darcs repository] so I ended up cloning the ones I wanted in the time that it took for the +++<cite>+++sync-all+++</cite>+++ script to run. ~~~text darcs get http://darcs.haskell.org/packages/base/ # gets most of the packages we'd be interested in darcs get http://darcs.haskell.org/packages/array/ # gets the array package ~~~

Of course I could have just been patient and waited for the script to finish\...
TextMate Bundles location on Mountain Lion
Date: 2012-12-31 23:59:42

Something that I've noticed when trying to install https://github.com/swannodette/textmate-clojure[various different] https://github.com/textmate/haskell.tmbundle[bundles] is that the installation instructions which worked flawlessly on Snow Leopard don't seem to do the job on Mountain Lion.

For example, the Clojure bundle assumes that the installation directory is '~/Library/Application\ Support/TextMate/Bundles' but for some reason the 'Bundles' folder doesn't exist.

We therefore have two choices:

* mkdir -p ~/Library/Application\ Support/TextMate/Bundles and then continue as normal
* Install our bundle into '/Applications/TextMate.app/Contents/SharedSupport/Bundles' http://stackoverflow.com/questions/4547076/textmate-haskell-bundle[as suggested on this thread].</p>

So for the Clojure bundle we'd do this instead:

[source,text]
----

$ cd /Applications/TextMate.app/Contents/SharedSupport/Bundles
$ git clone git://github.com/swannodette/textmate-clojure.git Clojure.tmbundle
$ osascript -e 'tell app "TextMate" to reload bundles'
----

And similarly for the Haskell one:

[source,text]
----

$ cd /Applications/TextMate.app/Contents/SharedSupport/Bundles
$ git clone https://github.com/textmate/haskell.tmbundle.git haskell.tmbundle
$ osascript -e 'tell app "TextMate" to reload bundles'
----

Thinking about it now I'm wondering whether I did actually create the 'Bundles' folder in '~/Library/Application\ Support/TextMate/' on my old machine and I just can't remember doing so!
Haskell: Using qualified imports to avoid polluting the namespace
Date: 2012-12-30 23:16:48

In most of the Haskell code I've read any functions from other modules have been imported directly into the namespace and I reached the stage where I had this list of imports in a file:

[source,haskell]
----

import System.IO
import Data.List.Split
import Data.Char
import Data.Bits
import Control.Monad
import Data.Map
import Data.Set
import Data.List
import Data.Maybe
----

This becomes a problem when you want to use a function which is defined in multiple modules such as +++<cite>+++filter+++</cite>+++:

[source,text]
----

clustering.hs:53:43:
    Ambiguous occurrence `filter'
    It could refer to either `Data.List.filter',
                             imported from `Data.List' at clustering.hs:11:1-16
                             (and originally defined in `GHC.List')
                          or `Data.Set.filter',
                             imported from `Data.Set' at clustering.hs:10:1-16
                          or `Data.Map.filter',
                             imported from `Data.Map' at clustering.hs:9:1-16
----

One way to solve this is to change occurrences of +++<cite>+++filter+++</cite>+++ to +++<cite>+++Data.List.filter+++</cite>+++ but it's a bit long winded and in this case there is a function in the +++<cite>+++Prelude+++</cite>+++ package which is available without us importing anything.

Unfortunately we'd have to use the prefix +++<cite>+++Prelude+++</cite>+++ to refer to it since all the other versions of the function have made it ambiguous.

We still want to use some of the functions in those other modules though so we can do a qualified import which will make the functions available to us but only if we refer to them by their full name. It won't import them into our namespace.

For example to initialise a map we'd do this:

[source,haskell]
----

> import qualified Data.Map
> Data.Map.assocs $ Data.Map.fromList [(1,2), (3,7)]
[(1,2),(3,7)]
----

That's a bit long winded though so we can rename imports with a shorter name to make our life a bit easier:

[source,haskell]
----

import System.IO
import Data.List.Split
import Data.Char
import Data.Bits
import qualified Control.Monad as Monad
import qualified Data.Map as Map
import qualified Data.Set as Set
import qualified Data.List as List
import qualified Data.Maybe as Maybe
----

We can then use functions in those packages like so:

[source,haskell]
----

> Maybe.maybeToList (Just 3)
[3]
----

For this particular function I haven't come across any with the same name so we might want to import that one into our namespace but require the use of the +++<cite>+++Maybe+++</cite>+++ prefix for any other functions:

[source,haskell]
----

import qualified Data.Maybe as Maybe
import Data.Maybe (maybeToList)
----

[source,haskell]
----

> maybeToList (Just 3)
[3]
----

There's a wiki entry I came across which http://en.wikibooks.org/wiki/Haskell/Modules#Renaming_imports[explains Haskell modules this in a bit more detail] and is worth a read.
Haskell: Pattern matching a list
Date: 2012-12-30 22:39:16

As I mentioned http://www.markhneedham.com/blog/2012/12/29/haskell-initialising-a-map/[in a post yesterday] I've been converting a clustering algorithm into Haskell and I wanted to get the value from doing a bit wise or on two values in a list.

I forgot it was possible to pattern match on lists until I http://www.markhneedham.com/blog/2012/04/15/haskell-a-simple-parsing-example-using-pattern-matching/[came across a post I wrote about 8 months ago] where I'd done this so my initial code looked like this:

[source,haskell]
----

> import Data.Bits
> map (\pair -> (pair !! 0) .|. (pair !! 1)) [[1,2], [3,4]]
[3,7]
----

We can pattern match against the list like so:

[source,haskell]
----

> map (\(x:y:_) -> x .|. y) [[1,2], [3,4]]
[3,7]
----

Here +++<cite>+++x+++</cite>+++ takes the first value, +++<cite>+++y+++</cite>+++ takes the second value and the rest of the list is assigned to +++<cite>+++_+++</cite>+++ which we don't use in this case.

There are loads of examples of pattern matching against different data structures in http://learnyouahaskell.com/syntax-in-functions[Learn You A Haskell] and hopefully next time I'll remember and won't write hideous code like the first example!
The Tracer Bullet Approach: An example
Date: 2012-12-24 09:09:44

A few weeks ago my former colleague https://twitter.com/kief[Kief Morris] wrote a blog post describing http://kief.com/tracer-bullet.html[the tracer bullet approach] he's used to setup a continuous delivery pipeline on his current project.

____
The idea is to get the simplest implementation of a pipeline in place, prioritizing a fully working skeleton that stretches across the full path to production over a fully featured, final-design functionality for each stage of the pipeline.
____

Kief goes on to explain in detail how we can go about executing this and it reminded of a project I worked on almost 3 years ago where we took a similar approach.

We were building an internal application for an insurance company and didn't have any idea how difficult it was going to be to put something into production so we decided to find out on the first day of the project.

We started small - our initial goal was to work out what the process would be to get a 'Hello world' text file onto production hardware.

Although we were only putting a text file into production we wanted to try and make the pipeline as similar as possible to how it would actually be so we set up a script to package the text file into a ZIP file. We then wired up a continuous integration server to generate this artifact on each run of the build.

What we learnt from this initial process was how far we'd be able to automate things. We were working closely with one of the guys in the operations team and he showed us where we should deploy the artifact so that he could pick it up and put it into production.

Our next step after this was to do the same thing but this time with a web application just serving a 'Hello world' response from one of the end points.

This was relatively painless but we learnt some other intricacies of the process when we wanted to deploy a script that would make changes to a database.

Since these changes had to be verified by a different person they preferred it if we put the SQL scripts in a different artifact which they could pick up.

We found all these things out within the first couple of weeks which made our life much easier when we put the application live a couple of months down the line.

Although there were a few manual steps in the process I've described *we still found the idea of driving out the path to production early a useful exercise*.

Read http://kief.com/tracer-bullet.html[Kief's post] for ideas about how to handle some of the problems you'll come across when it's all a bit more automated!
Weka: Saving and loading classifiers
Date: 2012-12-12 00:04:42

In our continued machine learning travels https://twitter.com/jennifersmithco[Jen] and I have been building some classifiers using http://www.cs.waikato.ac.nz/ml/weka/[Weka] and one thing we wanted to do was save the classifier and then reuse it later.

There is http://weka.wikispaces.com/Saving+and+loading+models[documentation] for how to do this from the command line but we're doing everything programatically and wanted to be able to save our classifiers from Java code.

As it turns out it's not too tricky when you know which classes to call and saving a classifier to a file is as simple as this:

[source,java]
----

MultilayerPerceptron classifier = new MultilayerPerceptron();
classifier.buildClassifier(instances); // instances gets passed in from elsewhere

Debug.saveToFile("/path/to/weka-neural-network", classifier);
----

If we want to load that classifier up we can make use of the +++<cite>+++http://weka.sourceforge.net/doc.dev/weka/classifiers/misc/SerializedClassifier.html[SerializedClassifier]+++<cite>+++class like so:</p> ~~~java SerializedClassifier classifier = new SerializedClassifier(); classifier.setModelFile(new File("/path/to/weka-neural-network")); ~~~

http://www.youtube.com/watch?v=Hl545RF6dXA[Simples]! +++</cite>++++++</cite>+++