markhneedham/blog huggingface dataset:
Haskell: Viewing the steps of a reduce
Date: 2012-02-25 23:40:07

I've been playing around with Haskell a bit over the last week and in the bit of code I was working on I wanted to fold over a collection but see the state of the fold after each step.

I remembered Don Syme showing me how to do something similar during the F# Exchange last year while we were http://fssnip.net/8R[writing some code to score a tennis game] by using +++<cite>+++http://msdn.microsoft.com/en-us/library/ee340364.aspx[Seq.scan]+++</cite>+++.

I didn't realise there was also a scan function in Haskell which could be defined in terms of +++<cite>+++foldl+++</cite>+++ if we wanted to:

[source,haskell]
----

foldscanl :: (a -> b -> a) -> a -> [b] -> [a]
foldscanl fn acc ls = foldl (\ x y -> x ++ [fn (last x) y]) [acc] ls
----

[source,haskell]
----

*Main> foldscanl (+) 0 [1..10]
[0,1,3,6,10,15,21,28,36,45,55]
----

The http://hackage.haskell.org/packages/archive/base/latest/doc/html/src/GHC-List.html#scanl[actual function is defined like so]:

[source,haskell]
----

scanl                   :: (a -> b -> a) -> a -> [b] -> [a]
scanl f q ls            =  q : (case ls of
                                []   -> []
                                x:xs -> scanl f (f q x) xs)
----

[source,haskell]
----

*Main> scanl (*) 1 [1..10]
[1,1,2,6,24,120,720,5040,40320,362880,3628800]
----

I've been finding +++<cite>+++scanl+++</cite>+++ particularly useful for working out whether the function I'm using to fold over the collection is actually working correctly.

There's also a +++<cite>+++scanr+++</cite>+++ if we want to fold over the collection from the right hand side.
Jersey: com.sun.jersey.api.client.ClientHandlerException: A message body reader for Java class [...] and MIME media type application/json was not found
Date: 2012-11-28 06:03:55

We've used the http://jersey.java.net/[Jersey] library on the last couple of Java based applications that I've worked on and one thing we've done on both of them is write services that communicate with each other using JSON.

On both occasions we didn't quite setup the Jersey client correctly and ended up with an error along these lines when making a call to an end point:

[source,java]
----

com.sun.jersey.api.client.ClientHandlerException: A message body reader for Java class java.util.ArrayList, and Java type java.util.ArrayList<com.blah.Message>, and MIME media type application/json was not found
! at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:561)
! at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:535)
! at com.sun.jersey.api.client.WebResource.handle(WebResource.java:696)
! at com.sun.jersey.api.client.WebResource.access$300(WebResource.java:74)
! at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:512)
...
! at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
! at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
! at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
! at java.lang.reflect.Method.invoke(Method.java:601)
! at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
! at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
! at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
! at com.yammer.metrics.jersey.InstrumentedResourceMethodDispatchProvider$TimedRequestDispatcher.dispatch(InstrumentedResourceMethodDispatchProvider.java:34)
! at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
! at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
! at com.sun.jersey.server.impl.uri.rules.ResourceObjectRule.accept(ResourceObjectRule.java:100)
! at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
! at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesR
----

To get around this problem we need to make sure we've added the 'JacksonJsonProvider' to the Jersey client config like so:

[source,java]
----

DefaultClientConfig defaultClientConfig = new DefaultClientConfig();
defaultClientConfig.getClasses().add(JacksonJsonProvider.class);
Client client = Client.create(defaultClientConfig);
----

I'm pretty sure that's documented somewhere in the depths of the Jersey wiki but since we've now ended up debugging this problem twice I thought it was worth writing down!
Micro Services: The curse of code 'duplication'
Date: 2012-11-28 08:11:04

A common approach we've been taking on some of the applications I've worked on recently is to decompose the system we're building into smaller http://2012.33degree.org/talk/show/67[micro services] which are independently deployable and communicate with each other over HTTP.

An advantage of decomposing systems like that is that we could have separate teams working on each service and then make use of a http://martinfowler.com/articles/consumerDrivenContracts.html[consumer driven contract] as a way of ensuring the contract between them is correct.

Often what actually happens is that we have one team working on all of the services which can lead to the a mentality where we treat start treating the comunication between services as if it's happening in process.

One of the earliest lessons I learnt when writing code is that you should http://en.wikipedia.org/wiki/Don't_repeat_yourself[avoid repeating yourself] in code - if you have two identical bits of code then look to extract that into a method somewhere and then call it from both locations.

This lesson often ends up getting applied across micro service boundaries when we have the same team working on both sides.

For example if we have a customer that we're sending between two services then in Java land we might create a +++<cite>+++CustomerDTO+++</cite>+++ in both services to marshall JSON to/from the wire.

We now have two versions of the 'same' object although that isn't necessarily the case because the client might not actually care about some of the fields that get sent because its definition of a customer is different than the provider's.

Nevertheless if we're used to being able to working with tools like http://www.jetbrains.com/idea/[IntelliJ] which let us make a change and see it reflected everywhere we'll end up driving towards a design where the +++<cite>+++CustomerDTO+++</cite>+++ is shared between the services.

This can be done via a JAR dependency or using something like http://git-scm.com/book/en/Git-Tools-Submodules[git sub modules] but in both cases we've now coupled the two services on their shared message format.

I think the 'duplication' thing might be less of an issue if you're using a language like Clojure where you could work with maps/lists without transforming them into objects but I haven't built anything web related with Clojure so I'm not sure.

As I understand it when we go down the micro services route we're trading off the ease of working with everything in one process/solution for the benefits of being able to deploy, scale and maintain parts of it independently.

Perhaps the problem I've described is just about getting used to this trade off rather than holding onto the idea that we can still treat it as a single application.

I'd be curious to hear others' experiences but I've seen this same pattern happen two or three times so I imagine it may well be common.
Buy vs Build: Driving from the problem
Date: 2012-11-17 16:56:28

My colleague https://twitter.com/erikdoe[Erik Doernenburg] has written a couple of articles recently discussing the reasons http://erik.doernenburg.com/2012/09/buy-vs-build-shift-part-1/[why people buy] and http://erik.doernenburg.com/2012/11/buy-vs-build-shift-part-2-and-3/[build IT solutions] and one part in particular resonated with me:

____
it is also possible, and not uncommon, that the software package does not do exactly what the business needs, leading to decreased productivity and lost opportunities.
____

I feel like there's a mindset change once you start thinking which package you could buy to solve your problem whereby *you stop solving the problem you actually have and focus instead on what features the package offers*.

Package software almost by definition tends to be written for a generic case of a problem and as Erik points out might not solve the problem that a business has. In addition it will probably solve problems which they don't have.

This leads to a tendency to believe that the alternative to buying a package is to match that package feature for feature with a custom built solution.

If you look at things from that perspective it's easy to see why buying would be preferred since the build approach is going to cost at least as much and you'd have to wait for it to be built as opposed to having it straight away.

However, when you start from the mindset of solving the problem in front of you - which is the mindset that building something leads you towards - then *you may actually find that you can get away with a much simpler implementation than previously seemed possible*.

By making use of prioritised iterative delivery we can then ensure that the most important/highest priority features are built first and if at any stage it doesn't make sense to build the next feature you can stop.

I'm curious whether there's some middle ground where you could still drive your solution from the problem and then pick and choose buy/build wherever it made sense.

We do actually take a variation of that approach in most applications I've worked on where we make use of open source libraries rather than writing everything from scratch.

In order for that approach to work with package software the package would need to be written more as a library that solves a specific problem rather than a framework which you need to use to solve every possible problem you have.

I don't really have a good conclusion to this but Erik covers this point and more in his http://erik.doernenburg.com/2012/09/buy-vs-build-shift-part-1/[two] http://erik.doernenburg.com/2012/11/buy-vs-build-shift-part-2-and-3/[posts] so go and read those if you find it interesting!
Unix: Counting the number of commas on a line
Date: 2012-11-10 16:30:48

A few weeks ago I was playing around with some data stored in a CSV file and wanted to do a simple check on the quality of the data by making sure that http://unix.stackexchange.com/questions/18736/how-to-count-the-number-of-a-specific-character-in-each-line[each line had the same number of fields].

One way this can be done is with +++<cite>+++awk+++</cite>+++:

[source,text]
----

awk -F "," ' { print NF-1 } ' file.csv
----

Here we're specifying the file separator +++<cite>+++-F+++</cite>+++ as ',' and then using the +++<cite>+++NF+++</cite>+++ (number of fields) variable to print how many commas there are on the line.

Another slightly more complicated way is to combine +++<cite>+++tr+++</cite>+++ and +++<cite>+++awk+++</cite>+++ like so:

[source,text]
----

tr -d -c ",\n" < file.csv | awk ' { print length } '
----

Here we're telling +++<cite>+++tr+++</cite>+++ to delete any characters except for a comma or new line.

If we pass just a comma to the '-d' option like so...

[source,text]
----

tr -d "," < file.csv
----

...that would delete all the commas from a line but we can use the '-c' option to complement the comma i.e. delete everything except for the comma.
[source,text]
----

tr -d -c "," < file.csv
----

Unfortunately that puts all the commas onto the same line so we need to complement the new line character as well:

[source,text]
----

tr -d -c ",\n" < file.csv
----

We can then use the +++<cite>+++length+++</cite>+++ variable of awk to print out the number of commas on each line.

We can achieve the same thing by making use of +++<cite>+++sed+++</cite>+++ instead of +++<cite>+++tr+++</cite>+++ like so:

[source,text]
----

sed 's/[^,]//g' file.csv | awk ' { print length } '
----

Since sed operates on a line by line basis we just need to tell it to substitute anything which isn't a comma with nothing and then pipe the output of that into awk and use the +++<cite>+++length+++</cite>+++ variable again.

I thought it might be possible to solve this problem using +++<cite>+++cut+++</cite>+++ as well but I can't see any way to get it to output the total number of fields.

If anyone knows any other cool ways to do the same thing let me know in the comments - it's always interesting to see how different people wield the unix tools!
IntelliJ Debug Mode: Viewing beyond 100 frames/items in an array
Date: 2012-11-26 04:28:22

In my continued attempts at the link:[Kaggle Digit Recognizer] problem I've been playing around with the http://www.heatonresearch.com/wiki/Encog_Java_Examples[encog] library to try and build a neural networks solution to the problem.

Unfortunately it's not quite working at the moment so I wanted to debug the code and see whether the input parameters were being correctly translated from the CSV file.

Each input is an array containing 784 values but by default IntelliJ restricts you to seeing 100 elements which wasn't helpful in my case since the early values tend to all be 0 and it's not until you get half way through that you see different values:

image::{{<siteurl>}}/uploads/2012/11/100limit.jpg[100limit,171]

It's http://blogs.jetbrains.com/idea/2011/09/debugger-working-with-sub-ranges-for-arrays-and-lists/[actually relatively easy] to see the other values by right clicking on the array in question and clicking 'Adjust Range':

image::{{<siteurl>}}/uploads/2012/11/Screen-shot-2012-11-25-at-22.59.42.jpeg[Screen shot 2012 11 25 at 22 59 42,307]

Then I type in the end index of the array and we're done!

image::{{<siteurl>}}/uploads/2012/11/Screen-shot-2012-11-25-at-23.00.44.jpeg[Screen shot 2012 11 25 at 23 00 44,388]
Java: java.lang.UnsupportedClassVersionError - Unsupported major.minor version 51.0
Date: 2012-11-24 08:49:28

On my current project we've spent the last day or so setting up an environment where we can deploy a couple of micro services to.

Although the machines are Windows based we're deploying the application onto a http://vagrantup.com/[vagrant] managed VM since the production environment will be a flavour of Linux.

Initially I was getting quite confused about whether or not we were in the VM or not and ended up with this error when trying to run the compiled JAR:

[source,text]
----

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/whatever/SomeService : Unsupported major.minor version 51.0
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClassCond(Unknown Source)
        at java.lang.ClassLoader.defineClass(Unknown Source)
        at java.security.SecureClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.access$000(Unknown Source)
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
Could not find the main class: com/whatever/SomeService. Program will exit.
----

These error means that you http://stackoverflow.com/questions/11239086/java-lang-unsupportedclassversionerror-unsupported-major-minor-version-51-0[compiled the code with a higher JDK than the one you're now trying to run it against].

Since I was accidentally trying to run the JAR against our Windows environment's 1.6 JDK rather than the VM's 1.7 JDK this is exactly what I was doing!

Muppet!
Windows line endings: Exception in thread 'main' java.io.FileNotFoundException /opt/app/config.yml{caret}M (no such file or directory)
Date: 2012-11-24 09:04:17

As I http://www.markhneedham.com/blog/2012/11/24/java-java-lang-unsupportedclassversionerror-unsupported-major-minor-version-51-0/[mentioned in my previous post] we've been making it possible to deploy our application to a new environment and as part of this we defined an http://upstart.ubuntu.com/[upstart] script which would run the JAR.

We tend to edit code on Windows and then test it out on the vagrant VM afterwards.

The end of our upstart script looked a bit like this:

[source,text]
----

script
    cd /opt/app
    java -jar /opt/app/app.jar /opt/app/config.yml
end script
----

Unfortunately when we tried to launch the application using 'start app' we got this error:

[source,text]
----

Exception in thread "main" java.io.FileNotFoundException /opt/app/config.yml^M (no such file or directory)
----

We were trying to load the configuration file in the program which was failing because of the Windows line ending just after the file name. That was being read in as part of our config file name argument.

I tried change the upstart script to have the name in quotes (which in hindsight makes no sense) but that made no difference but eventually I realised we could http://sourceforge.net/p/notepad-plus/discussion/331754/thread/ac70ed25[make the file have UNIX line endings] and solve our problem.

Using notepad++:

[source,text]
----

Edit > EOL Conversions > Unix Format
----

And all was well with the world.
Web Operations: Feature flags to turn off failing parts of infrastructure
Date: 2012-11-13 12:19:30

On most of the projects I've worked on over the last couple of years we've made use of http://martinfowler.com/bliki/FeatureToggle.html[feature toggles] that we used to turn pending features on and off while they were still being built but while reading http://www.amazon.co.uk/Web-Operations-Keeping-Data-ebook/dp/B0043M4Z34/ref=sr_1_2?ie=UTF8&qid=1352804969&sr=8-2[Web Operations] I came across another usage.

In the chapter titled 'Dev and Ops Collaboration and Cooperation' https://twitter.com/ph[Paul Hammond] suggests the following:

____
Eventually some of your infrastructure will fail in an unexpected way. When that happens, you'll want the ability to disable just the features that rely on it, and keep the rest of the site running. Feature flags make this possible.
____

We'd mainly use this approach to disable peripheral functionality such as the ability to comment on a site whose main purpose is to deliver news.

From what I understand this means we'd permanently have if statements (or some equivalent) in the appropriate places in our code base which could be dynamically toggled if we start experiencing problems.

This differs slightly from the feature toggle approach we've used because those toggles would eventually be removed when the feature was running successfully in production.

Hammond goes on to suggest using feature flags for any external services that we rely on e.g. Flickr relies on the Yahoo address book, del.icio.us and last.fm but can gracefully disable that functionality if needs be.

He also points out that it's useful to think hard about what features are absolutely core to serving your site e.g. Flickr can disable photo uploads but still allow people to continue viewing photos.

Overall this sounds like a pretty neat idea and apart from the slight complexity of having the conditionals in the code I can't really think of any reasons why you wouldn't want to do it. Happy to hear opposing views though!
Mahout: Using a saved Random Forest/DecisionTree
Date: 2012-10-27 22:03:30

One of the things that I wanted to do while http://www.markhneedham.com/blog/2012/10/27/kaggle-digit-recognizer-mahout-random-forest-attempt/[playing around with random forests] using http://mahout.apache.org/[Mahout] was to save the random forest and then use use it again which is something Mahout does cater for.

It was actually much easier to do this than I'd expected and assuming that we already have a +++<cite>+++https://github.com/apache/mahout/blob/trunk/core/src/main/java/org/apache/mahout/classifier/df/DecisionForest.java[DecisionForest]+++</cite>+++ built we'd just need the following code to save it to disc:

[source,java]
----

int numberOfTrees = 1;
Data data = loadData(...);
DecisionForest forest = buildForest(numberOfTrees, data);

String path = "saved-trees/" + numberOfTrees + "-trees.txt";
DataOutputStream dos = new DataOutputStream(new FileOutputStream(path));

forest.write(dos);
----

When I was looking through the API for how to load that file back into memory again it seemed like all the public methods required you to be using Hadoop in some way which I thought was going to be a problem as I'm not using it.

For example the signature for +++<cite>+++DecisionForest.load+++</cite>+++ reads like this:

[source,java]
----

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;

public static DecisionForest load(Configuration conf, Path forestPath) throws IOException { }
----

As it turns out though you can just pass an empty configuration and a normal file system path and the forest shall be loaded:

[source,java]
----

int numberOfTrees = 1;

Configuration config = new Configuration();
Path path = new Path("saved-trees/" + numberOfTrees + "-trees.txt");
DecisionForest forest = DecisionForest.load(config, path);
----

Much easier than expected!
Retrospectives: An alternative safety check
Date: 2012-10-27 18:21:57

At the start of most of the retrospectives I've been part of we've followed the safety check ritual whereby each person participating has to write a number from 1-5 on a sticky describing how they'll be participating in the retrospective.

1 means you'll probably keep quiet and not say much, 5 means you're perfectly comfortable saying anything and the other numbers fall in between those two extremes.

In my experiences it's a bit of a fruitless exercise because its viewed that a higher number is 'better' and therefore the minimum people will tend to write down is '3' because they don't want to stand out or cause a problem.

I've been in retrospectives where the majority of people were a '4' but then when it came to a contentious topic only 1 or 2 people contributed so I've become a bit sceptical of this safety check approach!

Before a retrospective I was facilitating a couple of months ago https://twitter.com/philandstuff[Phil] told me about an alternative that he preferred called ESVP which is described in http://www.amazon.co.uk/Agile-Retrospectives-Making-Pragmatic-Programmers/dp/0977616649/ref=sr_1_1?ie=UTF8&qid=1351360514&sr=8-1[Agile Retrospectives] like so:

* **E**xplorers are eager to discover new ideas and insights. They want to learn everything they can about the iteration/release/project.
* **S**hoppers will look over all the available information, and will be happy to go home with one useful new idea.
* **V**acationers aren't interested in the work of the retrospective, but are happy to be away from the daily grind. They may pay attention some of the time, but are mostly glad to be out of the office.
* **P**risoners feel that they've been forced to attend and would rather be doing something else.

Phil pointed out that in this case each of the possible categories describes how people will contribute to the retrospective whereas the traditional safety check is more about their degree of comfort with the environment.

I don't have enough data points to declare that this is better than the 1-5 approach but from my experiences it seems that there's less pressure on people to fit in one of the categories since none is 'better' than the other, just different.

In a recent retrospective where we used this it led to a conversation where people explained why they had chosen a particular category, which I thought was quite fascinating and something I haven't seen happen with the numbers approach.

My instinct is that if someone actually felt like a 'Prisoner' then they might not actually categorise themselves as that but I haven't seen this approach used enough to say that for certain!