markhneedham/blog huggingface dataset:
Scala: Prettifying test builders with package object
Date: 2011-07-26 22:31:58

We have several different http://www.natpryce.com/articles/000769.html[test builders] in our code base which look roughly like this:

[source,scala]
----

case class FooBuilder(bar : String, baz : String) {
	def build = new Foo(bar, baz)
}
----

In our tests we originally used them like this:

[source,scala]
----

class FooPageTest extends Specs with ShouldMatchers {
	it("should let us load a foo") {
		when(databaseHas(FooBuilder(bar = "Bar", baz = "Bazz")))
		// and so on...
	}
}
----

This works well but we wanted our tests to only contain domain language and no implementation details.

We therefore started pulling out methods like so:

[source,scala]
----

class FooPageTest extends Specs with ShouldMatchers {
	it("should let us load a foo") {
		when(databaseHas(aFooWithBarAndBaz("Bar", "Bazza")))
		// and so on...
	}

	def aFooWithBarAndBaz(bar:String, baz;String) = FooBuilder(bar = bar, baz = baz)
}
----

This was fine to start with but we eventually ended up with 10-12 different variations oh how +++<cite>+++Foo+++</cite>+++ could be constructed, negating the value that the builder pattern provides.

Instead what we can do is use of an alias of +++<cite>+++FooBuilder+++</cite>+++ to achieve something equally readable:

[source,scala]
----

package object TestSugar {
	val aFooWith = FooBuilder
}
----

We can then use +++<cite>+++aFooWith+++</cite>+++ like so:

[source,scala]
----

import TestSugar._

class FooPageTest extends Specs with ShouldMatchers {
	it("should let us load a foo") {
		when(databaseHas(aFooWith(bar = "Bar", baz = "Bazza")))
		// and so on...
	}
}
----

We could also achieve that by renaming +++<cite>+++FooBuilder+++</cite>+++ to +++<cite>+++aFooWith+++</cite>+++ but that makes it much less discoverable whereas this solution lets us achieve both goals.

The package object approach isn't really needed - we could easily put those vals onto an Object or Class but they don't really seem to belong to any which is why we've gone for this approach.
Scala: Traits galore
Date: 2011-07-09 19:54:05

We recently came across a problem where we had some logic that we wanted to be used by two classes.

Our original thought was to pull it up into an abstract class which ended up looking like this:

[source,scala]
----

abstract class SomeArbitraryClass(root:xml.Node) {
  def unrelatedField1:String
  def unrelatedField2:String

  def startPage:String
  def endPage:String
  def pageRange = if(firstPage == lastPage) "page %s".format(firstPage) else "pages %s-%s".format(firstPage, lastPage)
}
----

Writing a test _link to scala test_ for the page logic helped us to see more clearly that the design was a bit awkward:

[source,scala]
----

class SomeArbitraryClassTest extends Spec with ShouldMatchers {
  it("should format page range") {
    val someArbitraryClass = new SomeArbitraryClass(<empty />) {
      def unrelatedField1 = null
      def unrelatedField2 = null
      def startPage = "1"
      def endPage = "2"
    }

    someArbitraryClass.pageRange should equal("pages 1-2")
  }
}
----

It seemed pretty weird to have to pass in an empty XML tag to the class when our test didn't care at all about XML.

Having +++<cite>+++unrelatedField1+++</cite>+++ and +++<cite>+++unrelatedField2+++</cite>+++ as null values further helped us to see that the page stuff is totally unrelated to the rest of this class.

We therefore pulled out a trait just for the three page related functions:

[source,scala]
----

trait PageAware {
  def startPage:String
  def endPage:String
  def pageRange = if(firstPage == lastPage) "page %s".format(firstPage) else "pages %s-%s".format(firstPage, lastPage)
}
----

That trait is much more cohesive than the original +++<cite>+++Content+++</cite>+++ class and our new test reflects that:

[source,scala]
----

class PageAwareTest extends Spec with ShouldMatchers {
  it("should format page range") {
    val pageAware = new PageAware() {
      def startPage = "1"
      def endPage = "2"
    }

    pageAware.pageRange should equal("pages 1-2")
  }
}
----

And we can pull that trait into the abstract class too:

[source,scala]
----

abstract class SomeArbitraryClass(root:xml.Node) extends PageAware {
  def unrelatedField1:String
  def unrelatedField2:String
}
----

I guess we could also just pull in +++<cite>+++PageAware+++</cite>+++ at the sub class level but since all of the sub classes need to be page aware this seems to make sense.

We seem to be pulling out traits quite frequently as they seem to be a really nice way to abstract a group of related functions.

Our design is starting to resemble the http://www.engineyard.com/blog/2010/let-them-code-cake/[cake layering pattern] suggested by Martin Emde as an approach for avoiding ActiveRecord objects becoming cluttered with logic in Rails land.
Clojure: Getting caught out by lazy collections
Date: 2011-07-31 21:40:35

Most of the work that I've done with Clojure has involved running a bunch of functions directly in the REPL or through Leiningen's +++<cite>+++run+++</cite>+++ target which led to me getting caught out when I created a JAR and tried to run that.

As I http://www.markhneedham.com/blog/2011/07/10/clojure-language-as-thought-shaper/[mentioned a few weeks ago] I've been rewriting part of our system in Clojure to see how the design would differ and a couple of levels down the Clojure version comprises of applying a map function over a collection of documents.

The code in question originally looked like this:

[source,lisp]
----

(ns aim.main (:gen-class))

(defn import-zip-file [zipFile working-dir]
  (let [xml-files (filter xml-file? (unzip zipFile working-dir))]
    (map import-document xml-files)))

(defn -main [& args]
  (import-zip-file "our/file.zip", "/tmp/unzip/to/here"))
----

Which led to absolutely nothing happening when run like this!

[source,text]
----

$ lein uberjar && java -jar my-project-0.1.0-standalone.jar
----

I originally assumed that I had something wrong in the code but my colleague Uday reminded me that collections in Clojure are lazily evaluated and there was nothing in the code that would force the evaluation of ours.

In this situation we had to wrap the +++<cite>+++map+++</cite>+++ with a +++<cite>+++http://clojure.github.com/clojure/clojure.core-api.html#clojure.core/doall[doall]+++</cite>+++ in order to force evaluation of the collection:

[source,lisp]
----

(ns aim.main (:gen-class))

(defn import-zip-file [zip-file working-dir]
  (let [xml-files (filter xml-file? (unzip zip-file working-dir))]
    (doall (map import-document xml-files))))

(defn -main [& args]
  (import-zip-file "our/file.zip", "/tmp/unzip/to/here"))
----

When we run the code in the REPL or through 'lein run' the code is being eagerly evaluated as far as I understand it which is why we see a different behaviour than when we run it on its own.

I also got caught out on another occasion where I tried to pass around a collection of input streams which I'd retrieved from a zip file only to realise that when the code which used the input stream got evaluated the ZIP file was no longer around!
Scala: Companion Objects
Date: 2011-07-23 11:57:44

One of the language features available to us in Scala which I think is having a big impact in helping us to make our code base easier to follow is the http://daily-scala.blogspot.com/2009/09/companion-object.html[companion object].

We've been using companion objects quite liberally in our code base to define factory methods for our classes.

As I mentioned in a previous post http://www.markhneedham.com/blog/2011/06/26/coding-light-weight-wrapper-vs-serialisationdeserialisation/[a lot of our objects are acting as wrappers around XML documents] and we've been pushing some of the data extraction from the XML into companion objects so that our classes can take in non XML values.

This means we can test the data extraction against the companion object and then create simpler tests against any other logic in the object because we don't have to create XML documents in each of our tests.

The following is an example of a +++<cite>+++Foo+++</cite>+++ object being constructed with data from an XML document:

[source,scala]
----

object Foo {
   def apply(element: Node) = {
    val bar = element.attribute("bar").get.head.text
    val baz = (element \\ "baz").text
    new Foo(bar, baz)
  }
}
----

There is also some other logic around how a collection of +++<cite>+++Foos+++</cite>+++ should be ordered and by using the companion object to parse the XML we can create a test with appropriate +++<cite>+++bar+++</cite>+++ and +++<cite>+++baz+++</cite>+++ values to test that.

[source,scala]
----

case class Foo(bar: String, baz:String) extends Ordered[Foo] {
   def compare(that: Foo) = {
     // logic to compare Foos
   }
}
----

Before we had the companion object we were putting the logic to create +++<cite>+++Foo+++</cite>+++ inside the object where it is created from which increased the complexity of that object and made it more difficult for people to read.

We've also been using this approach to build up http://code.google.com/p/selenium/wiki/PageObjects[page objects] representing sub sections of a page in our Web Driver tests and it seems to work quite nicely there as well.
Scala: Making it easier to abstract code
Date: 2011-07-23 12:05:05

A couple of months ago I attended http://www.markhneedham.com/blog/2011/05/11/xp-2011-michael-feathers-brutal-refactoring/[Michael Feathers' 'Brutal Refactoring' workshop] at XP 2011 where he opined that *developers generally do the easiest thing when it comes to code bases*.

More often than not this means adding to an existing method or existing class rather than finding the correct place to put the behaviour that they want to add.

Something interesting that I've noticed so far on the project I'm working on is that so far *we haven't been seeing the same trend*.

Our code at the moment is comprised of lots of little classes with small amounts of logic in them and I'm inclined to believe that Scala as a language has had a reasonable influence on that.

The following quote from 'http://soft.vub.ac.be/~tvcutsem/whypls.html[Why programming languages?]' sums it up quite well:

____
Sometimes the growing complexity of existing programming languages prompts language designers to design new languages that lie within the same programming paradigm, but with the explicit goal of minimising complexity and maximising consistency, regularity and uniformity (in short, conceptual integrity).
____

It's incredibly easy to pull out a new class in Scala and the amount of code required to do so is minimal which seems to be contributing to the willingness to do so.

At the moment nearly all the methods in our code base are one line long and the ones which aren't do actually stand out which I think psychologically makes you want to find a way to keep to the one line method pattern.

== Traits

As I've mentioned previously we've been http://www.markhneedham.com/blog/2011/07/09/scala-traits-galore/[pulling out a lot of traits] as well and the only problem we've had there is ensuring that we don't end up testing their behaviour multiple times in the objects which mix-in the trait.

I tend to pull traits out when it seems like there might be an opportunity to use that bit of code rather than waiting for the need to arise.

That's generally not a good idea but it seems to be a bit of a trade off between making potentially reusable code discoverable and abstracting out the wrong bit of code because we did it too early.

== Companion Objects

The fact that we have http://www.markhneedham.com/blog/2011/07/23/scala-companion-objects/[companion objects] in the language also seems to help us push logic into the right place rather than putting it into an existing class.

We often have companion objects which take in an XML node, extract the appropriate parts of the document and then instantiate a case class object.

== In Summary

There's no reason you couldn't achieve the same things in C# or Java but I haven't seen code bases in those languages evolve in the same way.

It will be interesting to see if my observations remain the same as the code base increases in size.
Scala: An attempt to eradicate the if
Date: 2011-07-12 22:50:40

In a http://www.markhneedham.com/blog/2011/07/09/scala-traits-galore/[previous post] I included a code sample where we were formatting a page range differently depending on whether the start page and end pages were the same.

The code looked like this:

[source,scala]
----

trait PageAware {
  def startPage:String
  def endPage:String
  def pageRange = if(firstPage == lastPage) "page %s".format(firstPage) else "pages %s-%s".format(firstPage, lastPage)
}
----

Looking at the if statement on the last line we were curious whether it would be possible to get rid of it and replace it with something else.

In Java we could use the ternary operator:

[source,java]
----

public class PageAware {
  public String pageRange() {
    return (firstPage == lastPage) ? String.format("page %s", firstPage) : String.format("pages %s-%s", firstPage, lastPage)
  }
}
----

The if/else statement in Scala is supposed to replace that as far as I understand but I think the ternary operator looks neater.

Beyond defining that we played around with some potential alternatives.

We could use a +++<cite>+++Map+++</cite>+++ to store the true and false values as keys:

[source,scala]
----

trait PageAware {
  def pageRange = Map(true -> "page %s", false -> "pages %s-%s")(firstPage == lastPage).format(firstPage, lastPage)
}
----

http://twitter.com/#!/uday_rayala[Uday] came up with a pattern matching solution which looks like this:

[source,scala]
----

trait PageAware {
  def pageRange = ((firstPage, lastPage) match { case (`firstPage`, `firstPage`) => "page %s" case _ => "pages %s-%s"}).format(firstPage, lastPage)
}
----

Unfortunately both of these solutions are significantly less readable than the if/else one so it seems like this is one of the situations where it doesn't actually make sense to get rid of it.
Scala: Pattern matching a pair inside map/filter
Date: 2011-07-12 22:42:49

More than a few times recently we've wanted to use pattern matching on a collection of pairs/tuples and have run into trouble doing so.

It's easy enough if you don't try and pattern match:

[source,scala]
----

> List(("Mark", 4), ("Charles", 5)).filter(pair => pair._2 == 4)
res6: List[(java.lang.String, Int)] = List((Mark,4))
----

But if we try to use pattern matching:

[source,scala]
----

List(("Mark", 4), ("Charles", 5)).filter(case(name, number) => number == 4)
----

We end up with this error:

[source,text]
----

<console>:1: error: illegal start of simple expression
       List(("Mark", 4), ("Charles", 5)).filter(case(name, number) => number == 4)
----

It turns out that we can only use this if we pass the function to filter using {} instead of ():

[source,scala]
----

> List(("Mark", 4), ("Charles", 5)).filter { case(name, number) => number == 4 }
res7: List[(java.lang.String, Int)] = List((Mark,4))
----

It was pointed out to me on the http://www.scala-lang.org/node/813[Scala IRC channel] that the reason for the compilation failure has nothing to do with trying to do a pattern match inside a higher order function but that it's not actually possible to use a case token without the {}.

____
[23:16] mneedham: hey - trying to understand how pattern matching works inside higher order functions. Don't quite get this code \-> https://gist.github.com/1079110 any ideas? [23:17] dwins: mneedham: scala requires that "case" statements be inside curly braces. nothing to do with higher-order functions [23:17] mneedham: is there anywhere that's documented or is that just a known thing? [23:18] mneedham: I expected it to work in normal parentheses [23:21] amacleod: mneedham, it's documented. Whether it's documented simply as "case statements need to be in curly braces" is another question
____

The first line of Section 8.5 'Pattern Matching Anonymous Functions' of the http://www.scala-lang.org/docu/files/ScalaReference.pdf[Scala language spec] proves what I was told:

____
Syntax: BlockExpr ::= '`{`' CaseClauses '`}
____

It then goes into further detail about how the anonymous function gets converted into a pattern matching statement which is quite interesting reading.
Retrospectives: The 4 L's Retrospective
Date: 2011-07-25 21:00:30

I facilitated the latest retrospective my team had last week and decided to try http://ebgconsulting.com/blog/the-4l%E2%80%99s-a-retrospective-technique/[The 4 L's technique] which I'd come across while browsing the 'retrospectives' tag on del.icio.us.

We had 4 posters around the room representing each of the L's:

* Liked
* Learned
* Lacked
* Longed for

I'm not really a fan of the majority of a retrospective being dominated by a full group discussion as http://www.markhneedham.com/blog/2010/11/15/retrospectives-my-first-time-facilitating/[many people aren't comfortable giving their opinions to that many people] and therefore end up not participating at all.

I've seen much more participation if the facilitator tries to encourage less vocal people to give their opinions and if the first part of the retrospective is done in smaller groups.

We therefore started in groups of three where people discussed the previous iteration and came up with ideas which they stuck under each section. That lasted for around 15 minutes.

After that we split into groups of about 5 - one for each of the L's - and each group spent 6/7 minutes grouping together the stickies and looking for any trends.

One member of each group then presented a summary of their section to the rest of the group and suggested what they thought the most important thing to discuss was.

Having gone around all of the groups we now had 30 minutes to discuss the 4 topics we'd identified. In fact two of them were the same so we only had 3!

My observation of this style of retrospective is that it seemed to achieve the goal of getting more people to participate. At least 2 or 3 people who have never spoken in one of our retrospectives before were giving their opinions to the whole group.

I was curious to see whether we'd cover all the topics that people wanted to discuss as I cut the whole group voting system which I've seen used in most retrospectives I've attended.

After we'd finished discussing the 3 main topics a couple of other points were raised which had both been on the 'longed for' wall.

We ended up just quickly agreeing to give these things a try for an iteration instead of having a prolonged discussion about the advantages/disadvantages of the idea.

Facilitating wise I think I could have been clearer with my instructions as people were a bit confused at times about what exactly they were supposed to be doing.

I think it's vital to get everyone in the group involved early on or they just zone out and their insight is lost.

I'd be interested in hearing other types of retrospectives people have run which allow you to do that.
Parsing XML from the unix terminal/shell
Date: 2011-09-03 23:42:11

I spent a bit of time today trying to put together a quick script which would allow me to grab story numbers from the commits in our Git repository and then work out which functional areas those stories were in by querying mingle.

Therefore I wanted to make a +++<cite>+++curl+++</cite>+++ request to the mingle and then pipe that result somewhere and run an xpath expression to get my element.

I didn't want to have to write code in another script file and then reference that file from the shell and in my search to achieve that I came across +++<cite>+++http://xmlstar.sourceforge.net/docs.php[XMLStarlet]+++</cite>+++ on http://stackoverflow.com/questions/29004/parsing-xml-using-unix-terminal[stackoverflow].

It's installable via mac ports:

[source,text]
----

sudo port install xmlstarlet
----

And I was then able to pipe the results of my mingle request and locate the following bit of XML:

[source,text]
----

<property type_description="Managed text list" hidden="false">
<name>Functional Area</name>
<value>Our Functional Area</value>
</property>
----

[source,text]
----

curl -s http://user:password@mingleurl:8888/api/v2/projects/project_name/cards/1.xml  | xmlstarlet sel -t -v "//property/name[. = 'Functional Area']/../value"
----

There's much more you can do with the command which is listed on the http://xmlstar.sourceforge.net/doc/xmlstarlet.txt[documentation page].
The 'window fixing' wall
Date: 2011-09-20 06:49:43

On my current project we have a wall where we keep track of 'window fixing' tasks - things that people want to fix in the code base but chose to defer until a later date.

Every now and then we take what's on the wall and prioritise it according to http://fabiopereira.me/blog/2009/09/01/technical-debt-retrospective/[Fabio Pereira's effort/pain matrix] so that we know which clean up tasks will provide the greatest value to the team.

While I think it's a nice way of getting a team understanding of technical debt I think it can lead to a *couple of problems which come with most attempts at group responsibility for something*.

By writing the task up on the wall we've effectively pushed the responsibility for keeping the code clean away from us and onto the 'team'.

It also seems to make it more acceptable to make a mess in the code because we've acknowledged that we've done that and either us or a team mate will fix it later.

In a way I suppose it's good that people are at least conscious that they're taking short cuts at times and we have a reasonable log of where those short cuts have been taken.

On the other hand, from my experience, when people are really motivated to fix a piece of code then they'll find the time/way to do that whether or not it's written up on the wall.

I think this is also a good thing even though the refactoring won't have been prioritised by the rest of the team.

Sometimes it's easier to go and fix something when you know what needs doing rather than deferring it and having to explain the problem to someone else later.

In summary I think the centralised wall is a good idea but not a complete replacement for people being diligent and taking care of the code base themselves.