markhneedham/blog huggingface dataset:
Retrospective: The 5 whys
Date: 2011-10-24 22:53:14

Last week my colleague http://twitter.com/#!/patforna[Pat Fornasier] ran our team's fortnightly retrospective and one of the exercises we did was 'the 5 whys'.

I've always wanted to see how the 5 why's would pan out but could never see how you could fit it into a normal retrospective.

Pat was able to do this by using the data gathered by an earlier timeline exercise where the team had to plot the main events that had happened over the last 6 months.

We ended up with 5 key areas and split into groups to explore those topics further.

http://en.wikipedia.org/wiki/5_Whys[Wikipedia describes the 5 whys like so]:

____
The 5 Whys is a questions-asking method used to explore the cause/effect relationships underlying a particular problem. Ultimately, the goal of applying the 5 Whys method is to determine a root cause of a defect or problem.
____

My group had to investigate the topic 'Why are we so obsessed with points?'.

These were some of my observations from the exercise:

* It's *very easy to lose focus* on the exercise and start talking about solutions or ideas when only a couple of whys have been followed. Pat suggested that this problem could be solved by having a facilitator who helps keep the discussion on track.
* We went down a *dead-end* a few times where our 5th why ended up being something quite broad which we couldn't do anything about. We ended up going back up the chain of whys to see whether we could branch off a different way on any of the and it was actually reasonably easy to think of other whys the further up you went.
* By going beyond surface reasons for things you actually end up with *much more interesting conversations* although I think it does also become a little bit more uncomfortable for people. For example we ended up discussing what 'minimum viable product' actually means for us and a couple of the group had a much different opinion to the product owner. It would have been interesting if we'd been able to continue the discussion for longer.
* For our particular topic we ended up discussing why the deadline we have was set when it was and couldn't really come up with any reason for why it couldn't be changed other than we'd been told it couldn't. It would have been more interesting *to have the people external to the team* who set the deadline so that we could understand if there was more to it.

I tried looking for a video to see a real life example of a 5 whys discussion being facilitated but I wasn't able to find one.

http://twitter.com/perrynfowler[Perryn] pointed me to a https://github.com/cucumber/cucumber/wiki/[chat log on the cucumber wiki] where Aslak asks the 5 whys to someone trying to articulate why they want to have a login feature in their application but I'd be interested in seeing more examples if anyone knows any.
Bash: Reusing previous commands
Date: 2011-10-13 19:46:20

A lot of the time when I'm using the bash shell I want to re-use commands that I've previously entered and I've recently learnt some neat ways to do this from my colleagues http://twitter.com/#!/tomduckering[Tom] and http://twitter.com/kief[Kief].

If we want to list the history of all the commands we've entered in a shell session then the following command does the trick:

[source,text]
----

> history
...
  761  sudo port search pdfinfo
  762  to_ipad andersen-phd-thesis.pdf
  763  vi ~/.bash_profile
  764  source ~/.bash_profile
  765  to_ipad andersen-phd-thesis.pdf
  766  to_ipad spotify-p2p10.pdf
  767  mkdir LinearAlgebra
----

If we want to execute any of those commands again then we can do that by entering +++<cite>+++![numberOfCommand+++</cite>+++. For example, to execute the last command on that list we'd do this:

[source,text]
----

> !767
mkdir LinearAlgebra
mkdir: LinearAlgebra: File exists
----

We can also search the history and execute the last command that matches the search by doing the following:

[source,text]
----

> !mk
mkdir LinearAlgebra
mkdir: LinearAlgebra: File exists
----

A safer way to do this would be to suffix that with +++<cite>+++:p+++</cite>+++ so the command gets printed to stdout rather than executed:

[source,text]
----

> !mk:p
mkdir LinearAlgebra
----

A fairly common use case that I've come across is to search for a file and then once you've found it open it in a text editor.

We can do this by using the +++<cite>+++!!+++</cite>+++ command which repeats the previously executed command:

[source,text]
----

> find . -iname "someFile.txt"
> vi `!!`
----

We can achieve the same thing by wrapping '!!' inside '$()' as well:

[source,text]
----

> find . -iname "someFile.txt"
> vi $(!!)
----

Sam Rowe has a cool post where he http://samrowe.com/wordpress/advancing-in-the-bash-shell/[goes into this stuff in even more detail].

I'm sure there are more tricks that I haven't learnt yet so please let me know if you know some!
Scala: Adding logging around a repository
Date: 2011-10-25 21:19:22

We wanted to add some logging around one of our repositories to track how many times users were trying to do various things on the application and came across http://yacode.blogspot.com/2011/07/dynamic-proxy-using-scala-implicit.html[a cool blog post explaining how we might be able to do this].

We ended up with the following code:

[source,scala]
----

class BarRepository {
  def all: Seq[Bar] = Seq()
  def find(barId:String) : Bar = Bar("myBar")
}
----

[source,scala]
----

class TrackService(barRepository:BarRepository) {
  def all : Seq[Bar] = {
    var bars = barRepository.all;
    println("tracking all bars");
    bars
  }
}
----

[source,scala]
----

implicit def trackServiceToBarRepository(t:TrackService) : BarRepository = t.barRepository
----

We can then use it like this:

[source,scala]
----

scala> val service = new TrackService(new BarRepository())
service: TrackService = TrackService@4e5394c

scala> service.all
tracking all bars
res6: Seq[Bar] = List()
----

If a method doesn't exist on +++<cite>+++TrackService+++</cite>+++ then the implicit conversion ensures that the appropriate method will be called on +++<cite>+++BarRepository+++</cite>+++ directly:

[source,scala]
----

scala> service.find("mark")
res7: Bar = Bar(myBar)
----

I came across another way to achieve the same results by http://java.dzone.com/articles/real-world-scala-managing-cros[making use of traits] although we'd need to change our design a little bit to achieve this pattern:

[source,scala]
----

trait IProvideBars {
  def all : Seq[Bar]
  def find(barId:String) : Bar
}
----

[source,scala]
----

class BarRepository extends IProvideBars {
  def all: Seq[Bar] = Seq()
  def find(barId:String) : Bar = Bar("myBar")
}
----

[source,scala]
----

trait Tracking extends IProvideBars {
  abstract override def all : Seq[Bar] = {
    val bars = super.all;
    println("tracking all bars");
    bars
  }
}
----

[source,scala]
----

scala> val b = new BarRepository() with Tracking
b: BarRepository with Tracking = $anon$1@ddc652f

scala> b.all
tracking all bars
res8: Seq[Bar] = List()
----
Scala: Creating an Xml element with an optional attribute
Date: 2011-10-25 20:38:52

We have a lot of Xml in our application and one of the things that we need to do reasonably frequently in our test code is create elements which have optional attributes on them.

Our simple first approach looked like this:

[source,scala]
----

def createElement(attribute: Option[String]) = if(attribute.isDefined) <p bar={attribute.get} /> else <p />
----

That works but it always seemed like we should be able to do it in a simpler way.

Our first attempt was this:

[source,scala]
----

def createElement(attribute: Option[String]) = <p bar={attribute} />
----

But that ends up in a compilation error:

[source,text]
----

error: overloaded method constructor UnprefixedAttribute with alternatives:
  (key: String,value: Option[Seq[scala.xml.Node]],next: scala.xml.MetaData)scala.xml.UnprefixedAttribute <and>
  (key: String,value: String,next: scala.xml.MetaData)scala.xml.UnprefixedAttribute <and>
  (key: String,value: Seq[scala.xml.Node],next1: scala.xml.MetaData)scala.xml.UnprefixedAttribute
 cannot be applied to (java.lang.String, Option[String], scala.xml.MetaData)
       def createElement1(attribute: Option[String]) = <p bar={attribute} />
----

We really need to extract the string value from the option if there is one and not do anything if there isn't one but with the above approach we try to shove an option in as the attribute value. Unfortunately there isn't an overload of the constructor which lets us do that.

Eventually one of my colleagues suggested we try passing null in as the attribute value if we had a None option:

[source,scala]
----

def createElement(attribute: Option[String]) = <p bar={attribute.getOrElse(null)} />
----

Which works pretty well:

[source,text]
----

scala> createElement(Some("mark"))
res0: scala.xml.Elem = <p bar="mark"></p>


scala> createElement(None)
res1: scala.xml.Elem = <p ></p>
----
Clojure: Equivalent to Scala's flatMap/C#'s SelectMany
Date: 2011-07-03 22:50:47

I've been playing around with Clojure a bit over the weekend and one thing I got stuck with was working out how to achieve the functionality provided by Scala's +++<cite>+++flatMap+++</cite>+++ or C#'s +++<cite>+++SelectMany+++</cite>+++ methods on collections.

I had a collection of zip files and wanted to transform that into a collection of all the file entries in those files.

If we just use +++<cite>+++map+++</cite>+++ then we'll end up with a collection of collections which is more difficult to deal with going forward.

In Scala we'd do the following:

[source,scala]
----

import scala.collection.JavaConversions._

val zip1 = new ZipFile(new File("/Users/mneedham/Documents/my-zip-file.zip"))
val zip2 = new ZipFile(new File("/Users/mneedham/Documents/my-zip-file2.zip"))

List(zip1, zip2).flatMap(_.entries)
----

I was originally make using of +++<cite>+++map+++</cite>+++ followed by +++<cite>+++flatten+++</cite>+++ but I learnt from my colleague http://fragmental.tw/[Phil Calcado] that the function I wanted is +++<cite>+++http://clojure.github.com/clojure/clojure.core-api.html#clojure.core/mapcat[mapcat]+++</cite>+++ which leads to this solution:

[source,lisp]
----

(def zip1 (new ZipFile (file "/Users/mneedham/Documents/my-zip-file.zip")))
(def zip2 (new ZipFile (file "/Users/mneedham/Documents/my-zip-file2.zip")))

(mapcat (fn [file] (enumeration-seq (.entries file))) (list zip1 zip2))
----

I also learnt about the various functions available to create sequences, such as +++<cite>+++enumeration-seq+++</cite>+++ from other types which are listed at http://clojure.org/sequences[the bottom of this page].

Scala uses implicit conversions to do that and presumably you'd hide away the conversion in a helper function in Clojure.
Clojure: Creating XML document with namespaces
Date: 2011-07-20 20:28:17

As I mentioned in an earlier post we've been http://www.markhneedham.com/blog/2011/07/16/clojure-extracting-child-elements-from-an-xml-document-with-zip-filter/[parsing XML documents with the Clojure zip-filter API] and the next thing we needed to do was create a new XML document containing elements which needed to be inside a namespace.

We wanted to end up with a document which looked something like this:

[source,text]
----

<root>
<mynamespace:foo xmlns:mynamespace="http://www.magicalurlfornamespace.com">
	<mynamespace:bar>baz</mynamespace:bar>
</mynamespace:foo>
</root>
----

We can make use of +++<cite>+++lazy-xml/emit+++</cite>+++ to output an XML string from _some sort of input?_ by wrapping it inside +++<cite>+++with-out-str+++</cite>+++ like so:

[source,lisp]
----

(require '[clojure.contrib.lazy-xml :as lxml])
(defn xml-string [xml-zip] (with-out-str (lxml/emit xml-zip)))
----

I was initially confused about how we'd be able to create a map representing name spaced elements to pass to +++<cite>+++xml-string+++</cite>+++ but it turned out to be reasonably simple.

To create a non namespaced XML string we might pass +++<cite>+++xml-string+++</cite>+++ the following map:

[source,lisp]
----

(xml-string {:tag :root :content [{:tag :foo :content [{:tag :bar :content ["baz"]}]}]})
----

Which gives us this:

[source,text]
----

"<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<root>
	<foo>
		<bar>baz</bar>
	</foo>
</root>"
----

Ideally I wanted to prepend +++<cite>+++:foo+++</cite>+++ and +++<cite>+++:bar+++</cite>+++ with ':mynamespace" but I thought that wouldn't work since that type of syntax would be invalid in Ruby and I thought it'd be the same in Clojure.

[source,text]
----

mneedham@Administrators-MacBook-Pro-5.local ~$ irb
>> { :mynamespace:foo "bar" }
SyntaxError: compile error
(irb):1: odd number list for Hash
{ :mynamespace:foo "bar" }
               ^
(irb):1: syntax error, unexpected ':', expecting '}'
{ :mynamespace:foo "bar" }
               ^
(irb):1: syntax error, unexpected '}', expecting $end
	from (irb):1
>>
----

In fact it isn't so we can just do this:

[source,lisp]
----

(xml-string {:tag :root
  :content [{:tag :mynamespace:foo :attrs {:xmlns:meta "http://www.magicalurlfornamespace.com"}
              :content [{:tag :mynamespace:bar :content ["baz"]}]}]})
----

[source,text]
----

"<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<root>
<mynamespace:foo xmlns:meta=\"http://www.magicalurlfornamespace.com">
	<mynamespace:bar>baz</mynamespace:bar>
</mynamespace:foo>
</root>"
----

As a refactoring step, since I had to append the namespace to a lot of tags, I was able to make use of the +++<cite>+++keyword+++</cite>+++ function to do so:

[source,lisp]
----

(defn tag [name value] {:tag (keyword (str "mynamespace" name)) :content [value]})
----

[source,text]
----

> (tag :Foo "hello")
{:tag :mynamespace:Foo, :content ["hello"]}
----
Unix: Summing the total time from a log file
Date: 2011-07-27 23:02:33

As I http://www.markhneedham.com/blog/2011/07/27/a-crude-way-of-telling-if-a-remote-machine-is-a-vm/[mentioned in my last post] we've been doing some profiling of a data ingestion job and as a result have been putting some logging into our code to try and work out where we need to work on.

We end up with a log file peppered with different statements which looks a bit like the following:

[source,text]
----

18:50:08.086 [akka:event-driven:dispatcher:global-5] DEBUG - Imported document. /Users/mneedham/foo.xml in: 1298
18:50:09.064 [akka:event-driven:dispatcher:global-1] DEBUG - Imported document. /Users/mneedham/foo2.xml in: 798
18:50:09.712 [akka:event-driven:dispatcher:global-4] DEBUG - Imported document. /Users/mneedham/foo3.xml in: 298
18:50:10.336 [akka:event-driven:dispatcher:global-3] DEBUG - Imported document. /Users/mneedham/foo4.xml in: 898
18:50:10.982 [akka:event-driven:dispatcher:global-1] DEBUG - Imported document. /Users/mneedham/foo5.xml in: 12298
----

I can never quite tell which column I need to get so end up doing some exploration with awk like this to find out:

[source,text]
----

$ cat foo.log | awk ' { print $9 }'
1298
798
298
898
12298
----

Once we've worked out the column then we can add them together like this:

[source,text]
----

$ cat foo.log | awk ' { total+=$9 } END { print total }'
15590
----

I think that's much better than trying to determine the total run time in the application and printing it out to the log file.

We can also calculate other stats if we record a log entry for each record:

[source,text]
----

$ cat foo.log | awk ' { total+=$9; number+=1 } END { print total/number }'
3118
----

[source,text]
----

$ cat foo.log | awk 'min=="" || $9 < min {min=$9; minline=$0}; END{ print min}'
298
----
A crude way of telling if a remote machine is a VM
Date: 2011-07-27 22:31:20

We were doing a bit of profiling of a data importing process we've been running across various environments and wanted to check whether or not one of the environments was a physical machine or a VM.

A bit of googling first led me to http://www.coffer.com/mac_find/[the following site] where you can fill a MAC address and it will tell you which vendor it belongs to.

http://www.macvendorlookup.com[macvendorlookup.com] is even better though because it's more easily scriptable!

If I wanted to find the vendor of my MAC address on the ethernet port I could try the following:

[source,text]
----

ifconfig | grep -A1 en1 | grep ether | cut -d" " -f2 | xargs -I {} curl -s http://www.macvendorlookup.com/getoui.php?mac={} -o - | sed -e :a -e 's/<[^>]*>//g;/</N;//ba'
----

Which gives:

[source,text]
----

Vendor: Apple Inc
----

Sed magic was shamelessly stolen from http://www.eng.cam.ac.uk/help/tpl/unix/sed.html[sed one liners].

As it turns out the machine we wanted to learn about was a VM hosted on VMWare!
Performance tuning our data import: Gather precise data
Date: 2011-07-29 01:34:04

One of the interesting problems that we have to solve on my current project is working out how to import a few million XML documents into our database in a reasonable amount of time.

The stages of the import process are as follows:

. Extract a bunch of ZIP files to the disc
. Processing only the XML documents\...
. Load the XML document and determine whether the document is valid to import
. Add some meta data to the document for database indexing
. Import the document into the database

We've been working on this quite a bit recently and one of the main things we've learnt is the value of gathering detailing information about what's actually happening in the code.

When we started we only gathered the end to end time for the whole job to run against a certain number of documents.

The problem with doing this is that we couldn't see where the constraint in the process is and therefore went and parallelised the process using Akka which gave some improvement but not as much as expected.

Having realised that we didn't really know where the bottle neck was we added in much more logging to our code to try and identify where the most time was being taken.

For each document there are effectively 3 main things that we're doing:

* Loading the XML file
* Applying the XPath expressions against the file
* Importing the document into the database

image::{{<siteurl>}}/uploads/2011/07/performance.jpg[Performance,510]

We ran our import process a few times and recorded how long was being taken on each stage.

It was then much easier to see where we needed to focus our attention if we wanted to see big improvements.

We gathered this data for our local environment and QA environment and noticed that there was a big difference on the loading of the XML files - it was 6 or 7 times quicker on the QA environment.

By chance I ended up running the import on a laptop on the train and noticed that it aborted because it couldn't access an external DTD which was referenced in each XML file.

The QA machine is sitting inside a data centre with a high speed connection which means that the downloading of the DTD files is significantly faster than we can achieve locally.

We realised that we could solve this problem by forcing the parser to load the DTDs locally and immediately saw a huge decrease in the overall time.

Without collecting the data and seeing so clearly where the constraint was it would have taken us much longer to realise where we needed to make improvements.

We still have many more improvements to make but measuring the performance instead of speculating seems to be the way to go.
Emacs: Re-mapping the Control and Meta Keys on Mac OS X
Date: 2011-07-17 10:24:13

Since I've started playing around with Clojure again I thought it'd make sense to use emacs as my editor and therefore needed to work out how to remap the Ctrl and Meta to keys which are more accessible on the MBP's keyboard.

I've found that I like using the Caps Lock for Ctrl and that's reasonably easy to change by navigating to 'System Preferences > Keyboard > Modifier Keys':

image::{{<siteurl>}}/uploads/2011/07/ctrl-emacs.jpg[Ctrl emacs,473]

The Meta key is Escape by default and I wanted to map it to the right Command key as that's much easier for me to get to.

I couldn't find a way to do that through System Preferences but found  http://pqrs.org/macosx/keyremap4macbook/document.html[KeyRemap4MacBook] which seems to do the job.

We can change the key mapping by navigating to 'System Preferences > KeyRemap4Macbook > Change Key':

image::{{<siteurl>}}/uploads/2011/07/meta-emacs.jpg[Meta emacs,561]
Scala: Rolling with implicit
Date: 2011-07-19 06:39:44

We've been coding in Scala on my project for around 6 weeks now and are getting to the stage where we're probably becoming a big dangerous with our desire to try out some of the language features.

One that we're trying out at the moment is the http://www.scala-lang.org/node/114[implicit] key word which allows you to pass arguments to objects and methods without explicitly defining them in the parameter list.

The website we're working on needs to be accessible in multiple languages and therefore we need to be able to translate some words before they get displayed on the page.

Most of the time it's just static labels which need to be internationalised but there are a few words which are retrieved from the database and aren't as easy to deal with.

We introduced the idea of the +++<cite>+++LanguageAwareString+++</cite>+++ which acts as a wrapper around a +++<cite>+++String+++</cite>+++ and has its own +++<cite>+++toString+++</cite>+++ method which delegates to a +++<cite>+++Language+++</cite>+++ class which contains a dictionary

It's defined like this:

[source,scala]
----

case class LanguageAwareString(ignorantValue:String)(implicit val language : Language) {
  def toString = language.translate(ignorantValue)
}
----

We didn't want to have to pass +++<cite>+++Language+++</cite>+++ to the +++<cite>+++LanguageAwareString+++</cite>+++ factory method every time we're going to be calling it in quite a few places.

We therefore create an implicit val at the beginning of our application in the Scalatra entry code

[source,scala]
----

class Controllers extends ScalatraFilter with ScalateSupport {
	...
	implicit def currentLanguage : Language = // work out the current language
}
----

As I understand it, whenever the Scala compiler encounters an implicit it looks in its execution scope for any value defined as implicit with the expected type.

As long as there's only one such value in the scope it make use of that value but if there's more than one we'd see a compilation error since it wouldn't know which one to use.

We therefore needed to define +++<cite>+++Language+++</cite>+++ as an implicit on all the classes/methods which the code follows on its way down to +++<cite>+++LanguageAwareStrong+++</cite>+++.

The problem we've had is that it's not immediately obvious what's going on to someone who hasn't come across  +++<cite>+++implicit+++</cite>+++ before and we therefore end up having to go the above each time!

We've decided that to ease that transition we'd explicitly pass +++<cite>+++Language+++</cite>+++ down through the first few classes so that it's more obvious what's going on.

We therefore have code like this in a few places:

[source,scala]
----

new ObjectThatTakesLanguageImplicitly(someArg)(currentLanguage)
----

Maybe we can phase that out as people get used to +++<cite>+++implicit+++</cite>+++ or maybe we'll just get rid of +++<cite>+++implicit+++</cite>+++ and decide it's not worth the hassle!