markhneedham/blog huggingface dataset:
Coding: Light weight wrapper vs serialisation/deserialisation
Date: 2011-06-26 13:58:10

image::{{<siteurl>}}/uploads/2011/06/WrapperObjects.jpg[WrapperObjects,231]

As I've http://www.markhneedham.com/blog/2011/06/18/marklogic-deleting-all-the-documents-in-a-database/[mentioned] http://www.markhneedham.com/blog/2011/06/20/marklogic-customising-a-result-set/[before], we're making use of a MarkLogic database on the project I'm working on which means that we're getting quite big XML data structures coming into our application whenever we execute a query.

The normal way that I've seen for dealing with external systems would be to create an http://www.markhneedham.com/blog/2009/07/07/domain-driven-design-anti-corruption-layer/[anti corruption layer] where we initialise objects in our system with the required data from the external system.

In this case we've decided that approach doesn't seem to make as much sense because we don't need to do that much with the data that we get back.

We effectively map straight into a http://codebetter.com/gregyoung/2010/02/15/cqrs-is-more-work-because-of-the-read-model/[read model] where the only logic is some formatting for how the data will be displayed on the page.

The read model objects look a bit like this:

[source,scala]
----

class Content(root : xml.Node) {
    def numberOfResults: Int = (root \ "@count").text.toInt
}
----

They are just lightweight wrapper objects and we make use of Scala's XML support to retrieve the various bits of content onto the page.

The advantage of doing things this way is that it means we have less code to write than we would with the serialisation/deserialisation approach although it does mean that we're strongly coupled to the data format that our storage mechanism uses.

However, since this is one bit of the architecture which is not going to change it seems to makes sense to accept the leakage of that layer.

So far the approach seems to be working out fine but it'll be interesting to see how well it holds up if those lightweight wrappers do end up needing to have more logic in them.
IntelliJ: Adding resources with unusual extensions onto the classpath
Date: 2011-06-09 23:10:23

We're making use of http://www.marklogic.com/[MarkLogic] and therefore http://www.w3schools.com/xquery/default.asp[xquery] on the project I'm currently working on and recently wanted to add our xquery setup files onto the classpath so they could be used in a test.

We added them into 'src/main/resources' and set that as a source path in IntelliJ assuming that was all we needed to do.

Despite doing that our test kept failing because it couldn't locate the files on the classpath.

http://www.linkedin.com/profile/view?id=8818047&authType=name&authToken=w9yv&locale=en_US&pvs=pp&trk=ppro_viewmore[Charles] eventually came across the compiler resource patterns settings which are accessible from the Preferences > Compiler menu:

image::{{<siteurl>}}/uploads/2011/06/intellij.jpg[Intellij,434]

We had to add "*.xqy" to the end of the list of patterns to get our files picked up.

We came across another file extension that we needed to be on the classpath about 10 minutes later and obviously solved it much quicker that time!
Scala: Setting a default value
Date: 2011-06-12 16:03:30

We wanted to try and generate a build label to use for the name of the artifacts archive that gets generated each time we run the build but wanted to default it to a hard coded value if the system property representing the build label wasn't available.

In Ruby we would be able to do something like this:

[source,ruby]
----

buildLabel =  ENV["GO_PIPELINE_LABEL"] || "LOCAL"
----

There isn't a function in Scala that does that so we initially ended up with this:

[source,scala]
----

  def buildLabel() = [{
    System.getenv("GO_PIPELINE_LABEL") match {
      case null => "LOCAL"
      case label => label
    }
  }
----

My colleague http://twitter.com/#!/mushtaqA[Mushtaq] suggested passing the initial value into an Option like so...

[source,scala]
----

def buildLabel() = {
	Option(System.getenv("GO_PIPELINE_LABEL")).getOrElse("LOCAL")
}
----

...which I think is pretty neat!

I tried to see what the definition of an operator to do it the Ruby way would look like and ended up with the following:

[source,scala]
----

class RichAny[A](value:A ) {
      def || (default:A ) = {  Option(value).getOrElse(default)  }
}
----

[source,scala]
----

implicit def any2RichAny[A <: AnyRef](x: A) = new RichAny(x)
----

Which we can use like so:

[source,scala]
----

def buildLabel() = {
System.getenv("GO_PIPELINE_LABEL") || "LABEL"
}
----

I imagine that's probably not the idiomatic Scala way to do it so I'd be curious to know what is.
Parkinson's Law and Iteration Zero
Date: 2011-06-13 23:02:57

I've been thinking a bit about http://en.wikipedia.org/wiki/Parkinson's_Law[Parkinson's Law] recently and its' applicability in software development.

Parkinson's law is defined as follows:

____
Parkinson's Law is the adage first articulated by Cyril Northcote Parkinson as the first sentence of a humorous essay published in The Economist in 1955: "`Work expands so as to fill the time available for its completion`"
____

My colleagues quite frequently reference this law with respect to stories taking the amount of time that reflects the story point estimate assigned to them.

I haven't noticed this so much but I think we're more susceptible to the law when what we're working on doesn't have a clear goal or doesn't have a fast feedback loop.

One of the times where I think we run into this problem is in the agile 'iteration zero'.

Iteration Zero is an iteration reserved at the start of a project for setting up project infrastructure, building a walking skeleton and other such tasks. It's typically a week long.

While I think it's useful to do some up front work like this what I've noticed having participated in several of these is that we probably don't need a week for this type of work even though it will easily fill a week if it needs to.

Despite my belief that the work fills the time it's interesting to notice that we still don't get everything perfectly setup in iteration zero and probably wouldn't even if it was 2 weeks long instead.

We need to actually start driving some user functionality end to end and getting it deployed across our environments so that we can get proper feedback on the work that we've done.

For example one of the tasks of an iteration zero might be to have all the developer work stations ready to go by the time we start.

It's certainly possible to get the work stations to a stage where we think we've got everything setup correctly but it's not until people are actually using them that we know for sure.

The idea of an iteration zero is still useful but we should try to keep it as short as possible and accept that we're still going to be learning/spending time in the first few iterations on iteration zero type stuff and expect a 'slower velocity' accordingly.
Scala/Mustache: Creating a comma separated list
Date: 2011-06-22 21:24:06

We're using the http://scalate.fusesource.org/documentation/mustache.html[Mustache] templating engine on my project at the moment and one thing that we wanted to do was build a comma separated list.

Mustache is designed so that you pretty much can't do any logic in the template which made it really difficult to do what we wanted.

It's easy enough to get a comma after each item in a list with something like the following code:

[source,text]
----

{{#people}}<a href="/link/to/{{toString}}">{{toString}}</a>{{/people}}
----

where +++<cite>+++people+++</cite>+++ is passed to the template as a collection of strings.

To get rid of the trailing comma we ended up building a collection of +++<cite>+++Pairs+++</cite>+++ containing the person's name and a boolean value indicating whether or not to show the comma.

We need to show the comma before every element except for the first one so we can pass the following collection to the template:

[source,scala]
----

val values = names.zipWithIndex.map { case(item, index) => if(index == 0) (item, false) else (item, true) }
----

+++<cite>+++zipWithIndex+++</cite>+++ returns a collection of pairs containing the original strings and their position in the collection.

We can then map to a different result for just the first element and then use those pairs in the template like so:

[source,text]
----

{{#people}} {{#_2}}, {{/_2}}<a href="/link/to/{{_1}}">{{_1}}</a>{{/people}}
----

It's truly horrendous so if anyone knows a better way then please let me know!
Tech Leading: Keeping the passion
Date: 2011-06-22 23:48:08

As I http://www.markhneedham.com/blog/2011/04/17/tech-leading-initial-thoughts/[mentioned a couple of months ago], while I was in India I was acting as the Tech Lead on the project the TWU grads were working on and one thing I learnt from doing that is the importance of trying to keep the passion of the developers on the team.

When we started off I was more focused on trying to encourage the team to try and develop as many of the stories as possible.

This meant that on one occasion when one of the developers was really keen to try something out which he thought would make our life easier I overruled him in favour of working on a story that more directly contributed to what we were trying to do.

Luckily he didn't get dissuaded by me doing that and a few days later he again asked if he could explore that avenue as he was convinced it would be useful.

We decided that it would be worth trying for the rest of the day and it was noticeable that he was significantly more pumped up/enthusiastic on most things that he worked on after that.

I appreciate that I was in an unusual situation where the output of the project came second to the learning of the participants but I still feel like there is something to be said for helping encourage and keep the passion of  people on our teams.

It will mean sacrificing a bit of short term productivity, but if it means that the person has satisfied their curiosity about the viability of a bit of technology or even better has solved a problem for the team by doing so then it might be worth it.

As I found out it's much easier to knock someone back and you have justification to do so because what they want to do isn't directly on the value path of what's being paid for but that might not be the best decision if you look at the whole!

I'd be curious to hear the opinions of people who have played the tech lead role frequently because my observations are only based on a short stint and from being on the other side of the equation!
Scala: val, lazy val and def
Date: 2011-06-22 23:04:44

We have a variety of +++<cite>+++val+++</cite>+++, </cite>lazy val</cite> and +++<cite>+++def+++</cite>+++ definitions across our code base but have been led to believe that idiomatic Scala would have us using +++<cite>+++lazy val+++</cite>+++ as frequently as possible.

As far as I understand so far this is what the different things do:

* +++<cite>+++val+++</cite>+++ evaluates as soon as you initialise the object and stores the result.
* +++<cite>+++lazy+++</cite>+++ val evaluates the first time that it's accessed and stores the result.
* +++<cite>+++def+++</cite>+++ executes the piece of code every time - pretty much like a Java method would.

In Java, C# or Ruby I would definitely favour the 3rd option because it http://www.markhneedham.com/blog/2009/09/02/coding-reduce-fields-delay-calculations/[reduces the amount of state that an object has to hold].

I'm not sure that having that state matters so much in Scala because all the default data structures we use are immutable so you can't do any harm by having access to them.

I recently read http://www.codequarterly.com/2011/rich-hickey/[an interesting quote from Rich Hickey] which seems applicable here:

____
To the extent the data is immutable, there is little harm that can come of providing access, other than that someone could come to depend upon something that might change. Well, okay, people do that all the time in real life, and when things change, they adapt.
____

If the data was mutable then it would be possible to change it from any other place in the class which would make it difficult to reason about the object because the data might be in an unexpected state.

If we define something as a +++<cite>+++val+++</cite>+++ in Scala then it's not even possible to change the reference to that value so it doesn't seem problematic.

Perhaps I just require a bit of a mind shift to not worry so much about state if it's immutable.

It's only been a few weeks so I'd be interested to hear the opinions of more seasoned Scala users.

'''

I've read that there are various performance gains to be had from making use of +++<cite>+++lazy val+++</cite>+++ or +++<cite>+++def+++</cite>+++ depending on the usage of the properties but that would seem to be a premature optimisation so we haven't been considering it so far.
The supposed black box
Date: 2011-12-20 23:57:51

On a reasonable number of the systems that I've worked on over the past few years there's been a 'black box' component which the team I've been on has needed to integrate with.

I've always found it a little strange that you wouldn't need to/want to know how that part of the system worked or that you could actually believe that it was truly a black box.

If it doesn't work then you have no way of diagnosing the problem - did you do something wrong, was there something wrong inside the black box or was it something else.

On a project I worked on a few years ago the reason for the black box thinking was that http://www.markhneedham.com/blog/2009/04/14/lean-big-picture-over-local-optimisations/[each layer was being developed by people from a different company].

The problem we had was that we were working on the top layer, the one that was visible to the end user and therefore our progress was very visible to the stakeholders who were paying for the product to be built.

We therefore had no choice but to go into the metaphorical black box and try and gather as much information as we could to pass on to the teams working on the other layers so that they would be able to help us better.

I recently watched a talk by Artur Bergman titled 'http://www.youtube.com/watch?v=oebqlzblfyo[Full Stack Awareness]' where he talks about the necessity of understanding exactly what is happening when our code gets executed rather than thinking of it as magic.

Although Artur is working in a different context to most application developers who maybe don't need to know the stack as well as he does I think the advice about treating something as magic is useful.

If we think of something as a 'black box' then effectively we are saying that it's somewhat magic.

If the integrated component is being custom written then I think the team who needs to integrate with it should at the very least have someone who knows how it works very well so they can diagnose any problems quickly.

That person then needs to spread their knowledge amongst the rest of the team so that they don't end up being the bottle neck.

In summary I think the term 'black box' is frequently a misnomer and we'll rarely be able to view said black box in such an opaque way.
Learning Android: WebView character encoding
Date: 2011-12-27 23:53:56

In my continued attempts to learn how to write an Android application I came across a problem with character encoding when trying to load some text into a http://developer.android.com/reference/android/webkit/WebView.html[WebView].

I was initially trying to write the text to the WebView like this:

[source,java]
----

WebView webview = new WebView(collection.getContext());
webview.loadData(textWithQuotesIn, "text/html", "UTF-8");
----

image::{{<siteurl>}}/uploads/2011/12/android-before-after1.gif[Android before after,555]

But ended up with the output in the picture on the left hand side. I tried playing around with the encoding and debugged the application all the way through until it hit the WebView but there didn't seem to be any problem with the text.

I eventually http://stackoverflow.com/questions/7412763/string-encoding-problem-in-webview[came across a post on StackOverflow] where +++<cite>+++mice+++</cite>+++ suggested using one of the other methods available for writing to a WebView.

I changed my code to read like this:

[source,java]
----

WebView webview = new WebView(collection.getContext());
webview.loadDataWithBaseURL(url, textWithQuotesIn, "text/html", "UTF-8", url);
----

And now the single quotes are rendering correctly as can be seen on the image on the right.

I had a quick look at the Android source code to see if there was any obvious reason why one of the methods would work and the other wouldn't but I couldn't see anything.

Perhaps I'm doing something wrong with my call to 'loadData' and that's why it's not rendering the character set correctly. If that's the case please let me know.
Leiningen: Using goose via a local Maven repository
Date: 2011-12-27 12:48:17

I've been playing around a little bit with https://github.com/jiminoc/goose[goose] - a HTML content/article extractor - originally in Java but later in clojure where I needed to work out how to include goose and all its dependencies via https://github.com/technomancy/leiningen[Leiningen].

goose isn't included in a Maven repository so I needed to create a local repository, something which I've got stuck on in the past.

Luckily http://www.pgrs.net/2011/10/30/using-local-jars-with-leiningen/[Paul Gross has written a cool blog post] explaining how his team got past this problem.

Following the instructions from Paul's post this is how I got goose playing nicely with clojure:

Inside my clojure project:

[source,text]
----

/Users/mneedham/github/android/text-extraction $ mkdir maven_repository
----

I then ran the following command from where I had goose checked out on my machine:

[source,text]
----

mvn install:install-file -Dfile=target/goose-2.1.6.jar -DartifactId=goose -Dversion=2.1.6 -DgroupId=goose -Dpackaging=jar -DlocalRepositoryPath=/Users/mneedham/github/android/text-extraction/maven_repository -DpomFile=pom.xml
----

I added the repository and goose dependency to my project.clj file which now looks like this:

[source,text]
----

(defproject textextraction "0.1.0"
  :description "Extract text from urls"
  :dependencies [[org.clojure/clojure "1.2.0"],
		 [org.clojure/clojure-contrib "1.2.0"],
		 [ring/ring-jetty-adapter "0.3.11"],
         [compojure "0.6.4"]
         [goose "2.1.6"]]
  :dev-dependencies [[swank-clojure "1.2.1"]]
  :repositories {"local" ~(str (.toURI (java.io.File. "maven_repository")))}
  :main textextraction.main)
----

I then run:

[source,text]
----

/Users/mneedham/github/android/text-extraction $ lein run
----

And goose and all its dependencies are included in the 'lib' directory.
Reading Code: Know what you're looking for
Date: 2011-12-29 02:43:34

In the last week or so before Christmas I got the chance to spend some time pairing with my colleague http://www.linkedin.com/pub/alex-harin/13/40b/716[Alex Harin] while trying to understand how an existing application which we were investigating was written.

We knew from watching a demo of the application that the user was able to send some processing off to be done in the background and that they would be emailed once that had happened.

Our starting point was therefore to work backwards from the labels on the UI and finding which code got executed when the user submitted the task.

My initial approach was to find the entry point and then follow the method calls line by line, slowly building my knowledge of how the application actually worked.

Alex used a much quicker approach whereby he *thought about how the code would be designed* and then looked for the bit of code which proved his belief.

In this case we knew that the application had a 2 tier architecture and that it didn't use any middleware which meant that it would more than likely be using the database as a queue to store the tasks to be processed.

Another colleague and I were then able to make use of this approach when looking at another piece of code.

It was being used to do pricing and we knew that there were different algorithms depending on which country you were in, which meant that we needed to look for anything country related to answer our questions.

Effectively we're looking at the code with a hypothesis in hand and then trying to see whether or not what we see proves or disproves that hypotheses.

I need to practice a bit more but it seems to let you navigate code much more quickly and makes you much less likely to dive down a rabbit hole.
Clojure: Casting to a Java class...or not!
Date: 2011-12-31 17:47:47

I have a bit of Java code for http://stackoverflow.com/questions/2659000/java-how-to-find-the-redirected-url-of-a-url[working out the final destination of a URL] assuming that there might be one redirect which looks like this:

[source,java]
----

private String resolveUrl(String url) {
  try {
    HttpURLConnection con = (HttpURLConnection) (new URL(url).openConnection());
    con.setInstanceFollowRedirects(false);
    con.connect();
    int responseCode = con.getResponseCode();

    if (String.valueOf(responseCode).startsWith("3")) {
      return con.getHeaderField("Location");
    }
  } catch (IOException e) {
    return url;
  }

  return url;
}
----

I need to cast to +++<cite>+++HttpURLConnection+++</cite>+++ on the first line so that I can make the call to +++<cite>+++setInstanceFollowRedirects+++</cite>+++ which isn't available on +++<cite>+++URLConnection+++</cite>+++.

I wanted to write some similar code in Clojure and my first thought was that I needed to work out how to do the cast, which I didn't know how to do.

I then remembered that Clojure is actually dynamically typed so there isn't any need - as long as the object has the method that we want to call on it everything will be fine.

In this case we end up with the following code:

[source,clojure]
----

(defn resolve-url [url]
  (let [con (.. (new URL url) openConnection)]
    (doall
     (.setInstanceFollowRedirects con false)
     (.connect con))
    (if (.startsWith (str (.getResponseCode con)) "3")
      (.getHeaderField con "Location")
      url)))
----

Which can be simplified to this:

[source,clojure]
----

(defn resolve-url [url]
  (let [con (doto (.. (new URL url) openConnection)
                  (.setInstanceFollowRedirects false)
                  (.connect))]
  (if (.startsWith (str (.getResponseCode con)) "3")
    (.getHeaderField con "Location")
    url)))
----
Yak Shaving: Tracking the yak stack
Date: 2011-12-31 03:54:00

image::{{<siteurl>}}/uploads/2011/12/yak-stack.jpg[Yak stack,300]

While I've been http://www.markhneedham.com/blog/category/android-2/[learning how to write an android application] there's been plenty of opportunities for me to go off http://www.markhneedham.com/blog/?s=shaving+yak[shaving yaks], it's pretty much Yakville Central.

Typically I'd end up spending hours trying to work out some obscure thing which I didn't really need to know so I wanted to try and avoid that this time.

I started keeping a track of the 'yak stack' which I was currently following and mentally noting exactly where I was up to.

An example of a yak stack I kept while trying to authorise a user of the app with Twitter using OAuth is shown in the photo on the right hand side.

It ended up looking like this:

* Get the home timeline working
 ** OAuth blowing up
  *** Not actually capturing redirect back to app
   **** Launch mode in Android manifest

Once I realise I'm heading down the stack I've been giving myself one http://www.pomodorotechnique.com/[pomodoro] to  try and dig myself one level up.

If I still haven't managed to solve the problem I might keep going for one more pomodoro or just find another way around the problem.

I'm sure I'll come across problems where I need to spend more than an hour trying to solve it but for now it's working ok as a rule of thumb.

It's definitely fun chasing yaks but I get to the end of the day and haven't really achieved anything which isn't fun.