markhneedham/blog huggingface dataset:
Learning Regular Expressions: Non capturing match
Date: 2011-09-07 20:47:14

I've been working my way slowly through the O'Reilly 'http://www.amazon.co.uk/Mastering-Regular-Expressions-Jeffrey-Friedl/dp/0596528124/ref=sr_1_2?ie=UTF8&qid=1315428243&sr=8-2[Mastering Regular Expressions]' book and recently read about the non capturing match operator which came in useful for some Git log parsing I've been doing.

On the project I'm working on we all commit as the same user and then put our names at the beginning of the commit message.

We wanted to try and find out the statistics of who'd been pairing with each other and therefore needed to extract the pairs from commits.

Unfortunately everyone writes their names in a slightly different way so the regular expression which I used to parse each commit needed to try and handle that.

For example these are some of the ways that commit messages start:

[source,text]
----

Uday/Charles #67 did some stuff
mark,suzuki more stuff
pat, tom: very important stuff
Uday:Marc #87 stuff
----

The separator between the names is different in each case but in the majority of cases can be satisfied by the following regular expression:

[source,text]
----

([\/,][ ]?|:)
----

It's either:

* A forward slash or comma followed by an optional space
* A colon

Since I want to express the fact that the separator can be one thing or the other I need to group those two things together in parentheses.

Unfortunately that means that the separator will be included in the array of captures that we have when parsing the commit.

I only wanted to have the names of the two people included in that array.

The non capturing match operator '(?:' allows us to match against the expected separator without actually capturing it:

[source,text]
----

(?:[\/,][ ]?|:)
----

That regular expression is part of a much larger/probably over complicated one which also helps to capture the names of the people pairing:

[source,text]
----

var pairRegex = /^\[?([\w-]+)[ ]?[^\/, ]*(?:[\/,][ ]?|:)([\w-]+)\]?[^\/]*[\s:]/
----

Using the regex with the non capturing match gives us:

[source,text]
----

"charles/mark: adios to play, hello scalatra".match(pairRegex)
["charles/mark: adios to play, hello ", "charles", "mark"]
----

Whereas if we used a normal capture we'd also capture the '/':

[source,text]
----

"charles/mark: adios to play, hello scalatra".match(pairRegex)
["charles/mark: adios to play, hello ", "charles", "/", "mark"]
----
Retrospectives: Getting overly focused on actions
Date: 2011-09-24 06:56:39

I've attended a lot of different retrospectives over the last few years and one thing that seems to happen quite frequently is that a problem will be raised and there will become a massive urgency to find an action to match with that problem.

As a result of this we don't tend to go very deeply into working out why that problem happened in the first place and how we can stop it happening in the first place.

Any *discussion tends to be quite shallow and doesn't delve very far beyond the surface of the problem*.

I've noticed that this tends to happen more when there are a lot of people in the retrospective and there's a desire not to 'waste' everyone's time which is understandable to some extent.

We recently had an iteration where there were a lot of stories going back and forth between the developers and testers which was leading to a lot of context switching for some developers.

Since it had felt very disruptive we tried to find some way of deciding when we should or shouldn't context switch from the current story to fix bugs on earlier stories.

In hindsight it would have been more interesting to look at why that problem existed in the first place rather than directly addressing the problem.

In this case, as my colleague http://www.christopherbird.co.uk/[Chris] pointed out, it might make more sense for a developer (pair) to go and work with a tester on the story until it was ready to be signed off rather than switching back and forth.

I've read about other retrospective formats such as the 'http://www.nomad8.com/files/5why_root_cause_analysis_retrospective.php[five whys]' which might help a team to dig deeper into the problems they're facing but I'm curious whether it'd make sense to follow such a format with over 30 people attending.

We'd need to pick a sufficiently general problem to analyse so that everyone remained engaged.

I'd be curious whether anyone else has made a similar observation and how they made their retrospectives more effective.
Scala: for comprehensions with Options
Date: 2011-09-15 22:21:14

I've generally avoided using for expressions in Scala because the keyword reminds me of for loops in Java/C# and I want to learn to program in a less imperative way.

After working with my colleague http://twitter.com/#!/mushtaqA[Mushtaq] I realised that in some cases using for comprehensions can lead to much more readable code.

An interesting use case where this is the case is when we want to create an object from a bunch of parameters that may or may not be set. i.e. a bunch of options.

For example we might take some input from the user where they have to enter their name but could choose to leave the field blank:

[source,scala]
----

val maybeFirstName : Option[String] = Some("Mark")
val maybeSurname : Option[String] = None
----

We only want to create a +++<cite>+++Person+++</cite>+++ if they have provided both names.

The for comprehension works quite well in allowing us to do this:

[source,scala]
----

case class Person(firstName:String, surname:String)
----

[source,scala]
----

scala> for { firstName <- maybeFirstName; surname <- maybeSurname } yield Person(firstName, surname)
res27: Option[Person] = None
----

If we set the surname to have a value:

[source,scala]
----

val maybeSurname : Option[String] = Some("Needham")
----

Running the same for comprehension will yield a +++<cite>+++Person+++</cite>+++

[source,scala]
----

scala> for { firstName <- maybeFirstName; surname <- maybeSurname } yield Person(firstName, surname)
res29: Option[Person] = Some(Person(Mark,Needham))
----

From what I understand  when we have multiple values assigned using '\<-' inside a for comprehension, each value will have +++<cite>+++flatMap+++</cite>+++ called on it except for the last one which will have +++<cite>+++map+++</cite>+++ called instead.

The equivalent code if we didn't use a for comprehension would therefore look like this:

[source,scala]
----

scala> maybeFirstName.flatMap { firstName => maybeSurname.map { surname => Person(firstName, surname) } }
res43: Option[Person] = Some(Person(Mark,Needham))
----

For me the for comprehension expresses intent much better and it seems to excel even more as we add more values to the comprehension.
gawk: Getting story numbers from git commit messages
Date: 2011-09-12 07:05:13

As I mentioned in my previous post I've been writing a little application to create graphs based on our git repository history and in one of them we wanted to try and create a graph showing which people had been working on which stories.

I needed a way to extract a story number from the git commit message and then store them all in a text file.

A typical commit with a story number in might look like this:

[source,text]
----

Mark/Uday #689 some awesome scala refactoring
----

I couldn't think of an easy way to do this with my current knowledge of sed or the Mac version of awk but the +++<cite>+++http://www.gnu.org/software/gawk/manual/gawk.html#index-g_t_0040code_007bmatch_0028_0029_007d-function-1373[match]+++</cite>+++ function of gawk (GNU awk) makes this really easy.

____
match(string, regexp [, array]) Search string for the longest, leftmost substring matched by the regular expression, regexp and return the character position, or index, at which that substring begins (one, if it starts at the beginning of string). If no match is found, return zero. \... If array is present, it is cleared, and then the zeroth element of array is set to the entire portion of string matched by regexp.
____

The +++<cite>+++array+++</cite>+++ argument is what I needed and it's only available as a gawk extension according to the documentation.

I ended up with the following command to strip the story numbers:

[source,text]
----

git log --no-merges --pretty="format:%s" |
gawk '{ match($0, /#([0-9]+)/, arr); if(arr[1] != "") print arr[1] }'
----

I had to install gawk using ports on my Mac but on Fedora the default installation of awk is gawk.
Javascript: Internet Explorer 8 - trim() leads to 'Object doesn't support this property or method' error
Date: 2011-09-13 13:33:43

We make use of the Javascript +++<cite>+++https://developer.mozilla.org/en/JavaScript/Reference/Global_Objects/String/trim[trim()]+++</cite>+++ function in our application but didn't realise that it isn't implemented by Internet Explorer until version 9.

This led to the following error on IE8 when we used it:

____
Message: Object doesn't support this property or method Line: 18 Char: 13 Code: 0 URI: http://our.app/file.js
____

There's a http://stackoverflow.com/questions/2308134/trim-in-javascript-not-working-in-ie[stackoverflow thread] suggesting some different ways of implementing your own 'trim()' method but since we're using jQuery already we decided to just use the '$.trim()' function from there.

Therefore:

[source,javascript]
----

var cleaned = ourString.trim();
----

becomes:

[source,javascript]
----

var cleaned = $.trim(ourString);
----

I'm sure I must have come across this before but I don't remember when!
node.js: child_process.exec not returning all results
Date: 2011-09-22 19:55:45

I've been playing around with some node.js code to get each of the commits from our git repository but noticed that it didn't seem to be returning me all the results.

I had the following code:

[source,javascript]
----

var exec = require('child_process').exec;

var gitRepository = '/some/local/path';

exec('cd ' + gitRepository + ' && git log --pretty=format:"%H | %ad | %s%d" --date=raw ', function(error, stdout, stderror) {
  var commits = stdout.split("\n");

  // do some stuff with commits
});
----

We have around 2000 commits in the repository but I was only getting back 1600 of them when I checked the +++<cite>+++length+++</cite>+++ of +++<cite>+++commits+++</cite>+++.

Eventually I decided to print out what was in +++<cite>+++error+++</cite>+++ and got the following message:

[source,text]
----

error: Error: maxBuffer exceeded.
----

Going back to http://nodejs.org/docs/v0.4.8/api/all.html#child_process.exec[the documentation] revealed my mistake:

____
maxBuffer specifies the largest amount of data allowed on stdout or stderr - if this value is exceeded then the child process is killed. The default options are { encoding: 'utf8', timeout: 0, maxBuffer: 200*1024, killSignal: 'SIGTERM', cwd: null, env: null }
____

The limit is 2048000 which is around about the number of bytes being returned when I get to 1600 commits.

Changing the code to increase the buffer sorts it out:

[source,javascript]
----

var exec = require('child_process').exec;

var gitRepository = '/some/local/path';

exec('cd ' + gitRepository + ' && git log --pretty=format:"%H | %ad | %s%d" --date=raw ', {maxBuffer: 500*1024}, function(error, stdout, stderror) {
  var commits = stdout.split("\n");

  // do some stuff with commits
});
----
Scala: Replacing a trait with a fake one for testing
Date: 2011-09-25 10:24:20

We recently wanted to replace a trait mixed into one of our classes with a fake version to make it easier to test but forgot how exactly to do that!

The class is roughly like this:

[source,scala]
----

trait Foo { def foo : String = "real foo" }
class Mark extends Foo {}
----

We originally tried to replace it like this:

[source,scala]
----

trait BrokenFakeFoo { def foo : String = "broken fake foo" }
val m = new Mark with BrokenFakeFoo
----

[source,text]
----

error: overriding method foo in trait Foo of type => String;
 method foo in trait BrokenFakeFoo of type => String needs `override' modifier
       val m = new Mark with BrokenFakeFoo
----

If +++<cite>+++m+++</cite>+++ compiled it would have two versions of +++<cite>+++foo+++</cite>+++ but it wouldn't know which one to use, hence the error message.

Attempt two was this:

[source,scala]
----

trait BrokenFakeFoo { override def foo : String = "broken fake foo" }
----

[source,text]
----

error: method foo overrides nothing
       trait BrokenFakeFoo { override def foo : String = "broken fake foo" }
----

As https://github.com/uday-rayala[Uday] pointed out, what we actually need to do is make our fake trait extend the original one and then override the method.

[source,scala]
----

trait FakeFoo extends Foo { override def foo : String = "fake foo" }
val m = new Mark with FakeFoo
----

[source,text]
----

m.foo
> res5: String = fake foo
----

Since +++<cite>+++FakeFoo+++</cite>+++ is the right most of the traits mixed into +++<cite>+++Mark+++</cite>+++ its +++<cite>+++foo+++</cite>+++ method will be used over the +++<cite>+++Foo+++</cite>+++ one mixed into +++<cite>+++Mark+++</cite>+++ on its class definition.
jQuery: Collecting the results from a collection of asynchronous requests
Date: 2011-09-25 09:26:19

http://lizdouglass.wordpress.com/[Liz] and I recently spent some time building a pair stair to show how long ago people had paired with each other and one of the things we had to do was make AJAX requests to get the pairing data for each person and then collate it all to build the stair.

image::{{<siteurl>}}/uploads/2011/09/pair-stair.gif[Pair stair,545]

The original attempt to do this looked a bit like this:

[source,javascript]
----

var people = ["Marc", "Liz", "Ken", "Duncan", "Uday", "Mark", "Charles"];

var grid = [];
$.each(people, function(index, person) {
  $.getJSON('/git/pairs/' + person, function(data) {
    // parse data and create somethingCool
    grid.push(somethingCool);
  });
});

// do something with grid
----

When we try to do something with +++<cite>+++grid+++</cite>+++ it is of course empty because we've attempted to access it before all of the callbacks (which populate it) have returned.

Pedro Teixeira has http://metaduck.com/post/2675027550/asynchronous-iteration-patterns-in-node-js[a nice blog post which explains how to solve this problem in node.js] and we can use the same pattern here.

We need to write our own looping mechanism which is able to determine when the last callback has returned.

This is done by creating a copy of the +++<cite>+++people+++</cite>+++ array and then manually iterating through it using +++<cite>+++shift+++</cite>+++.

[source,javascript]
----

var people = ["Marc", "Liz", "Ken", "Duncan", "Uday", "Mark", "Charles"];
var peopleCopy = people.slice(0), grid = [];
(function getPairs() {
  var person = peopleCopy.shift();
	
  if(peopleCopy.length == 0) {
    // do something with grid
  } else {
    $.getJSON("/git/pairs" + person, function(data) {
      // parse data and create somethingCool
      grid.push(somethingCool);
      getPairs();		
    })						
  }
})();
----

I tried to extract the asynchronous looping and ended up with the following function:

[source,javascript]
----

function asyncLoop(collection, seedResult, loopFn, completionFn) {
  var copy = collection.slice(0);
  (function loop() {
    var item = copy.shift();
	
    if(copy.length == 0) {
      completionFn(seedResult);
    } else {
      loopFn(item, seedResult, loop);
    }	
  })();	
}
----

Which could be called like this:

[source,javascript]
----

var people = ["Marc", "Liz", "Ken", "Duncan", "Uday", "Mark", "Charles"];
asyncLoop(people, [], function(name, grid, callBackFn) {
  // parse data and create something cool
  grid.push(somethingCool);
  callBackFn();
}, function(grid) {
  // do something with grid
});
----

I'm not sure that it reads that much clearer but it does push some of the boiler plate code away.
The read-only database
Date: 2011-08-29 23:32:26

The last couple of applications I've worked on have had almost completely read only databases where we had to populate the database in an offline process and then provide various ways for users to access the data.

This creates an interesting situation with respect to how we should setup our development environment.

Our normal setup would probably have an individual version of that database on every development machine and we would populate and then truncate the database during various test scenarios.

image::{{<siteurl>}}/uploads/2011/08/test-data.jpg[Test data,320]

This actually means that our tests are interacting with the database in a different way than we would see during the running of the application.

It also means that we have more infrastructure to take care of and more software updates to do although using tools like Chef or Puppet can reduce the pain this causes once the initial setup of those scripts has been done.

On the project I worked on last year we started off with the individual database approach but eventually moved to having a shared database used by all the developers.

We only made the move once we had the real production data and the script which would populate that data into our database ready.

image::{{<siteurl>}}/uploads/2011/08/test-data-shared.jpg[Test data shared,320]

The disadvantage of having this shared database is that our tests become more indirect.

We wrote our tests against data which we knew would be in our production data set which meant if anything failed you had a bit more investigation to do since the data setup was done elsewhere.

On the other hand noone had to worry about getting it setup on their machines which had proved to be tricky to totally automate.

We have a similar situation on the application I'm currently working on and have noticed that we run into problems that don't usually exist as a result of adding data to the database on each test.

For example in one test the database takes a bit of time to sort out its indexes which means that some tests intermittently fail.

We found a bit of a hacky way around this by forcing the database to reindex in the test and waiting until it has done so but we've now solved a problem which doesn't actually exist in production.

This approach wouldn't work as well if we had a read/write database since we'd end up with tests failing since another developer machine had mutated the data it relied on.

With a read only database it seems to be ok though.
Scala, WebDriver and the Page Object Pattern
Date: 2011-08-09 00:54:23

We're using WebDriver on my project to automate our functional tests and as a result are using the http://code.google.com/p/selenium/wiki/PageObjects[Page Object pattern] to encapsulate each page of the application in our tests.

We've been trying to work out how to effectively reuse code since some of the pages have parts of them which work exactly the same as another page.

For example we had a test similar to this...

[source,scala]
----

class FooPageTests extends Spec with ShouldMatchers with FooPageSteps {
  it("is my dummy test") {
    ...
    then(iShouldNotSeeAnyCommonLinks())
  }
}
----

...where +++<cite>+++FooPageSteps+++</cite>+++ extends +++<cite>+++CommonSteps+++</cite>+++ which contains the common assertions:
[source,scala]
----

trait FooPageSteps extends CommonSteps {
  override val page = new FooPage(driver)
}
----

[source,scala]
----

trait CommonSteps {
  val page : FooPage
  val driver: HtmlUnitDriver

  def iShouldNotSeeAnyCommonLinks() {
    page.allCommonLinks.isEmpty should equal(true)
  }
}
----

+++<cite>+++FooPage+++</cite>+++ looks like this:

[source,scala]
----

class FooPage(override val driver:WebDriver) extends Page(driver) with CommonSection {

}

abstract class Page(val driver: WebDriver) {
  def title(): String = driver.getTitle;
}

trait CommonSection {
  val driver:WebDriver
  def allCommonLinks:Seq[String] = driver.findElements(By.cssSelector(".common-links li")).map(_.getText)
}
----

We wanted to reuse +++<cite>+++CommonSteps+++</cite>+++ for another page like so:

[source,scala]
----

trait BarPageSteps extends CommonSteps {
  override val page = new BarPage(driver)
}

class BarPage(override val driver:WebDriver) extends Page(driver) with CommonSection {

}
----

But that means that we need to change the type of +++<cite>+++page+++</cite>+++ in +++<cite>+++CommonSteps+++</cite>+++ to make it a bit more generic so it will work for +++<cite>+++BarPageSteps+++</cite>+++ too.

Making it of type +++<cite>+++Page+++</cite>+++ is not enough since we still need to be able to call the +++<cite>+++allCommonLinks+++</cite>+++ which is mixed into +++<cite>+++FooPage+++</cite>+++ by +++<cite>+++CommonSection+++</cite>+++.

We therefore end up with the following:

[source,scala]
----

trait CommonSteps {
  val page : Page with CommonSection
  val driver: HtmlUnitDriver

  def iShouldNotSeeAnyCommonLinks() {
    page.allCommonLinks.isEmpty should equal(true)
  }
}
----

We're able to mix in +++<cite>+++CommonSection+++</cite>+++ just for this instance of +++<cite>+++Page+++</cite>+++ which works pretty well for allowing us to achieve code reuse in this case!
Coding: The value in finding the generic abstraction
Date: 2011-08-31 06:49:48

I recently worked on adding the meta data section for each of the different document types that it serves which involved showing 15-20 pieces of data for each document type.

There are around 4-5 document types and although the meta data for each document type is similar it's not exactly the same!

When we got to the second document type it wasn't obvious where the abstraction was so we went for the copy/paste approach to see if it would be any easier to see the commonality if we put the two templates side by side.

We saw some duplication in the way that we were building up each individual piece of meta data but couldn't see any higher abstraction.

We eventually got through all the document types and hadn't really found a clean solution to the problem.

I wanted to spend some time playing around the code to see if I could find one but http://duncan-cragg.org/blog/[Duncan] pointed out that it was important to consider that refactoring in the bigger context of the application.

Even if we did find a really nice design it's probably not going to give us any benefit since we've covered most of the document types and there will maybe be just one that we have to add the meta data section for.

The return on investment for finding a clean generic abstraction won't be very high in this case.

In another part of our application we need to make it possible for the use to do http://en.wikipedia.org/wiki/Faceted_search[faceted search] but it hasn't been decided what the final list of facets to search on will be.

It therefore needs to be very easy to make it possible to search on a new facet and include details about that facet in all search results.

We spent a couple of days about 5/6 weeks ago working out how to model that bit of code so that it would be really easy to add a new facet since we knew that there would be more coming in future.

When that time eventually came last week it took just 2 or 3 lines of code to get the new facet up and running.

In this case spending the time to find the generic abstraction had a good return on investment.

I sometimes find it difficult to know exactly which bits of code we should invest a lot of time in because there are always loads of places where improvements can be made.

Analysing *whether there's going to be a future return on investment* from cleaning it up/finding the abstraction seems to be a useful thing to do.

Of course the return on investment I'm talking about here relates to the speed at which we can add future functionality.

I guess another return on investment could be reducing the time it takes to understand a piece of code if it's likely to be read frequently.