markhneedham/blog huggingface dataset:
Ask for forgiveness, not for permission
Date: 2010-06-04 21:03:38

I gave a presentation at our ThoughtWorks Brazil office in Porto Alegre last way on some of the things that I've learned while working at ThoughtWorks and the first point I made was that it was better to '*ask for forgiveness, not for permission*'.

This was something that was taught to me a few years ago and the idea behind this is that if there's some idea we want to try out it makes much more sense to start trying it now and then we can always apologise later on if someone has a problem with us doing that.

This is in preference to an approach where we ask whether we can try out an idea and get told no straight away i.e. we should look to be *proactive*.

It's really easy to say no to something when you don't really no what it is you're being asked and it seems to be the default answer a lot of the time.

An example of something where this rule may apply would be if we wanted to try writing some tests with a different language/framework to see whether it improves the way we're doing things.

We don't really know whether the new approach is going to be better than the current one so it makes sense to spend a bit of time playing around with it so that we can find out.

Later on in the talk I suggested that on some decisions it makes sense to gather data and then let the client decide what they want to do.

An example of this happened on a project I worked on a couple of years ago where the value of writing functional/unit tests was being questioned.

My colleague Kristan Vingrys came up with a matrix which showed the risks of not testing various features and then let the client decide whether they wanted to take that risk. http://blog.m.artins.net/measuring-test-effort/[Alex wrote more about this at the time].

After I mentioned this http://twitter.com/caiokf[Caio] pointed out that this idea seemed to contradict the idea of asking for forgiveness and not permission and at the time I couldn't really think of a good answer to why I didn't feel the situations were exactly the same.

A week on I think that the difference is that the 'ask for forgiveness' approach applies mainly for new situations where we haven't agreed as a team what we should do and people don't have enough information yet to know whether we should or should not do what is being proposed.

The idea of gathering data and letting the client decide applies more when a decision has been made or suggested and we want to try and influence that decision.

It doesn't make sense to just do what we want in this situation if it explicitly goes against what the client wants but equally we don't want to do something which goes against what we believe is the best approach without at least pointing that out.
Coding: Controlled Technical Debt
Date: 2010-06-20 22:37:32

A couple of months ago I wrote about http://www.markhneedham.com/blog/2010/04/14/agile-slimming-down-stories/[an approach to stories] that Christian has been encouraging on our project whereby we slim stories down to allow us to deliver the core functionality of the application as quickly as possible.

In our case we had a requirement to setup a range of different parameters used to lookup reference data used in the different calculations that we have in our application.

At the time we created an interface that the rest of the application would interact with so that we could easily substitute the real version in when we needed to:+++<center>+++image:{{<siteurl>}}/uploads/2010/06/technicaldebt.jpg[technicaldebt.jpg,514]+++</center>+++

We released the first version of the application about a month ago and finally last week implemented the story where the data would move from being in memory to being in a database table.

One of the requirements which we had delayed by only having these parameters in memory was the ability to easily modify them.

Any changes that needed to be made required an update of the code and then redeployment whereas with the database approach we would only have needed to deploy a delta script.

In the event there has only been one occasion so far where those parameters needed to be updated so it hasn't proved to be a problem.

Discussing this with a colleague on Friday he pointed out that what we'd done originally was to accept technical debt in our solution knowing that at some stage in the future we would need to address that.

The interesting thing about this case was that *we knew exactly when we were going to repay that debt* whereas it's often the case that we create technical debt in a code base and vaguely know that at some stage we'll address it.

As Martin Fowler points out in http://www.martinfowler.com/bliki/TechnicalDebt.html[his entry on technical debt]:

____
Just as a business incurs some debt to take advantage of a market opportunity developers may incur technical debt to hit an important deadline.
____

We probably saved at least a day's worth of effort by delaying this decision and were able to work on functionality that the business needed by the deadline instead. We then paid back that debt last week when we had more slack in the system.

The benefits of getting something into the market place quickly are much greater than I previously imagined and I think we can look at our assumptions of how a solution 'needs' to be designed much more closely to see if we can make these trade offs more frequently.
Slack time
Date: 2010-06-18 17:36:25

Ken Schwaber recently wrote a blog post where he compared http://kenschwaber.wordpress.com/2010/06/10/waterfall-leankanban-and-scrum-2/[the differences between the kanban, lean and scrum approaches to software development] and although I haven't had the same experiences as he has with the first two, one interesting thing he implies is that with a scrum approach we have slack time built in.

____
God help us. People found ways to have slack in waterfall, to rest and be creative. With Lean and Kanban, those hiding places are removed. We now have a progressive death march without pause.
____

The project that I'm currently working on has switched emphasis from being in pure delivery mode into a combination of delivery and handover and it's been quite noticeable how much more slack we have in the system as a result.

I find it much more enjoyable when there's a bit of slack such that we're not churning out story after story.

It gives you the opportunity to explore the code base a bit more and try out any ideas that you have to see if they're likely to improve the productivity of the team.

An example of this on our project is that we've had the time to introduce a calculation descriptions DSL that my colleague http://twitter.com/dermotkilroy[Dermot] had written in his own time.

This has helped reduce the number of tests required in certain areas of the code base as well as making the code more intuitive and therefore easy to understand.

It can be quite difficult to create this slack time when the team is under big deadline pressure and working on anything which isn't directly related to hitting that deadline isn't considered valuable.

I've noticed that the typical slack in the projects I've worked on tends to be towards the end of the day if we finish a story late on.

That tends to disappear if the team is working late and people start to become too tired to notice that there might be a better way to do things.

It seems to me that it would beneficial to try and work some slack time into projects to allow for the type of innovation that allows us to come up with ideas that allow us to be more productive.
NHibernate 2nd level cache: Doing it wrong?
Date: 2010-06-29 06:45:11

I wrote a couple of weeks ago about how we'd been trying to make use of the http://www.markhneedham.com/blog/2010/06/16/fluent-nhibernate-and-the-2nd-level-cache/[NHibernate 2nd level cache] and we were able to cache our data by following the various posts that I listed.

Unfortunately when we ran some performance tests we found that the performance of the application was significantly worse than when we just wrote our own 'cache' - an object which had a dictionary containing the reference data items we'd previously tried to lookup and the appropriate values.

We don't need to handle cache invalidation. The client's policy is to restart production servers every night so if we want to update any of the reference data then we just need to make sure a database script is run before the servers get restarted.

== Explicit transactions when reading

There is a post on the NHibernate Profiler website which describes why http://nhprof.com/Learn/Alerts/DoNotUseImplicitTransactions[we should not use implicit transactions] when using NHibernate. Instead we should create explicit ones:

____
When we don't define our own transactions, it falls back into implicit transaction mode, where every statement to the database runs in its own transaction, resulting in a large performance cost (database time to build and tear down transactions), and reduced consistency. Even if we are only reading data, we should use a transaction, because using transactions ensures that we get consistent results from the database.
____

We ended up with something like this:

[source,csharp]
----

public class OurRepository
{
	public ReferenceDataObject Find(ReferenceDataObjectKey key)
	{
		using(var session = SessionFactory.OpenSession())
		{
			using(var tx = session.BeginTransaction())
			{
				var query = session.Linq<ReferenceDataObject>().Where(r => r.Key == key);
				query.QueryOptions.SetCachable(true).SetCacheMode(CacheMode.Normal);
				// and so	 on
			}
		}

		// return the object
	}
}
----

As well as performance tests, we found that our integration tests became much slower than when we used our own 'cache'.

We have some tests which look up 100s of different bits of reference data and the total time taken to run those tests went from around 4 seconds up to 30 seconds.

As I understand it, putting a transaction around a query means that we create a transaction with the database on every request even if the query is cached and we're going to retrieve the results from the 2nd level cache rather than the database.

We took out the transaction which reduced the time taken to 7 seconds but it was still slower than when we used our hand rolled cache.

It seems like we must be doing something wrong or not understand something with respect to the NHibernate 2nd level cache because it seems ridiculous that the performance could be this different?
Intuition and 'quit thinking and look'
Date: 2010-06-28 08:39:10

Something which http://twitter.com/dermotkilroy[Dermot], http://twitter.com/christianralph[Christian] and I noticed last week is that on our project we've reached the stage where we intuitively know what the underlying problem is for any given error message in the application we're working on.

We're pretty much at the stage where we're effectively pattern matching what's going on without needing to think that much anymore.

This is a good thing because it saves a lot of time analysing every single message to try and work out what's going on - I think this means that we've reached a higher level of the http://www.markhneedham.com/blog/2009/08/10/dreyfus-model-more-thoughts/[Dreyfus model] when it comes to this particular situation.

The problem with getting too used to this approach is that eventually we'll come across a problem that we haven't come across before and if we don't recognise that this is the case we'll end up getting very frustrated.

I had the chance to work with http://www.oshineye.com/[Ade Oshineye] a few years ago and he always encouraged me to "*quit thinking and look*" when it came to problem solving.

This idea is derived from a book titled 'http://www.amazon.com/exec/obidos/ASIN/0814474578/debuggingrule-20[9 indispensable rules of debugging]' and the thinking behind it is that we often go straight to the solution for a problem without spending the time to understand what the problem actually is.

We actually came across a situation like this recently while investigating a problem in production.

We were getting an exception which looked pretty similar to other problems that we'd seen previously so we immediately tried that solution without any further investigation.

It had no impact at all so we had to go back and actually look at the error message we were receiving before trying something else.

As it turned out the time we wasted picking the wrong solution was less than 30 seconds but I think we had got to the point where a bit of *complacency had crept in* and we believed that there weren't any ways the application could go wrong that we hadn't seen before.

Talking further about this with Dermot he pointed out that this situation was akin to discovering an unknown unknown i.e. we came across a problem that we didn't know we didn't know about since we thought we knew about all of them!

Errol Morris wrote http://opinionator.blogs.nytimes.com/2010/06/20/the-anosognosics-dilemma-1/?hp[an interesting article about this recently in the New York Times] in which  referenced Dunning and Kruger's paper 'http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect[Unskilled and Unaware of It: How Difficulties of Recognizing One's Own Incompetence Lead to Inflated Self-assessments]'.

The learning for me from this is that while intuition is very useful it's also important to at least be aware that we probably don't know everything and that we may well come across new situations and will have to approach them as a novice again.
Using real life metaphors
Date: 2010-06-17 07:00:09

My colleague http://twitter.com/dermotkilroy[Dermot Kilroy] attended the http://skillsmatter.com/event/design-architecture/ddd-exchange-2010[DDD 2010 Exchange] in London last week and one of the ideas that he's been sharing with us from that is that of thinking how the user would solve a given problem without a technological solution i.e. how was something done before computers existed.

This encourages us to take a bigger picture view and can actually lead to a much simpler solution than we'd otherwise come up with.

An example that came up recently on the project I'm working on was related to how we should handle the situation where two agents tried to access the same users record at the same time. The agents are most likely in the same building.

The original technically focused solution was that we needed to lock the record so that if one agent had already opened the record then if another tried to access it they would be unable to.

Of course it's technically possible to do this but it's a bit tricky.

If there were no computers then there would probably be a physical folder which would contain all the information about the user.

It's not actually possible for two agents to both access that at the same time - if one wants to access it and it's already in use then they need not speak to their colleague and work out when they'll be finished with it.

If we consider it from this point of view then it becomes clear that putting in the effort to lock the record is unnecessary and that it would make more sense to just pop up a message on the screen informing the agent of the situation and allowing them to decide what to do.

This is the solution that we went with. It took way less time to implement and it's worked out fine so far.

---

I've retrospectively looked at this situation with my colleagues and realised that it seems to fit this pattern quite well but the way that it was reached originally was by http://www.markhneedham.com/blog/2010/03/26/finding-the-assumptions-in-stories/[looking at the requirement and finding the (initially hidden) reason for it].

It would be interesting to see whether we can rely on this degree of user integrity on a public facing application, I suspect that we can to a much greater degree than we might think.
Incremental Refactoring: Create factory method
Date: 2010-06-17 00:43:41

http://twitter.com/dermotkilroy[Dermot] and I spent a bit of time today refactoring some code where the logic had ended up in the wrong place.

The code originally looked a bit like this:

[source,csharp]
----

public class LookupService
{
	public LookUp Find(UserData userData)
	{
		var param1 = GetParam1From(userData);
		var param2 = GetParam2From(userData);
		var param3 = GetParam3From(userData);

		var lookupKey = new LookUpKey(param1, param2, param3);
		return lookupRepository.Find(lookupKey);
	}
}
----

[source,csharp]
----

public class LookUpKey
{
	private readonly string param1;
	private readonly string param2;
	private readonly string param3;

	public LookUpKey(string param1, string param2, string param3)
	{
		this.param1 = param1;
		this.param2 = param2;
		this.param3 = param3;
	}
}
----

We wanted to push the logic used to create the params in 'LookUpKey' into the 'LookUpKey' since it seemed that was a more natural place for that behaviour to live.

Unfortunately there were also a few other places that were use LookUpKey's constructor so we couldn't just go and change that to take in a 'UserData' otherwise we'd break other places in the code base.

My initial thought was that we could http://www.markhneedham.com/blog/2010/04/25/small-step-refactoring-overload-constructor/[overload the constructor] and temporarily have two constructors until we got rid of the original.

Dermot pointed out a better approach which was to add a static factory method to 'LookUpKey' and initially push the logic into that method.

[source,csharp]
----

public class LookUpKey
{
	...

	public static LookUpKey CreateFrom(UserData userData)
	{
		var param1 = GetParam1From(userData);
		var param2 = GetParam2From(userData);
		var param3 = GetParam3From(userData);

		return new LookUpKey(param1, param2, param3);
	}
}
----

The service is now much simplified:

[source,csharp]
----

public class LookupService
{
	public LookUp Find(UserData userData)
	{
		var lookupKey = LookUpKey.CreateFrom(userData);
		return lookupRepository.Find(lookupKey);
	}
}
----

We can also move tests that were originally indirectly testing the key creation by checking what was passed to the repository to be directly against the 'LookUpKey'.

Eventually we want to make LookUpKey's constructor private but for now we've been able to make our change without breaking the code base elsewhere.
Retrospectives: Some thoughts
Date: 2010-06-10 07:22:38

I've worked on two different teams this year which had quite different approaches to retrospectives.

In the first team we had a *retrospective at the beginning of every iteration* i.e. once every two weeks and in the second team we tried out the idea of having a *rolling retrospective* i.e. we put up potential retrospective items on the wall and when there were enough of those we discussed them in the standup.

The problem we had with the first approach was that we often had the *same types of discussions in each retrospective* and the same types of issues were raised each week so that it seemed almost pointless to even have it in the first place.

Several people ended up disliking the idea of even doing a retrospective in the first place because of this.

We came up with the idea of having a vote at the beginning of the iteration to decide whether or not we should have one with the idea being that if it didn't seem necessary then we could skip it for one week but would definitely then have the next one.

I think this worked reasonably well although it's slightly different to the approach taken by Rolf Knutsen which he discussed at link:[XP2010] in his session on 'Feedback and Retrospectives in Agile Projects'.

Rolf suggested that we should have a retrospective at a given time interval and just cut it short if we don't have anything else to discuss.

I think I prefer Rolf's suggestion as it's too easy to come to the conclusion that we don't need a retrospective especially if we think that they're a waste of time.

I thought the rolling retrospective approach that we took on the second project would work much better as we could just address any problems as soon as they came up but it hasn't worked quite that way from my experience.

Several issues which would normally be addressed in a retrospective seemed to go by unchecked and without being resolved.

Having a retrospective is a very useful way to *put people in a reflective mindset* where they look for things that aren't being done well or that could be done differently whereas during the day we might not do that.

I hadn't appreciated this benefit of a retrospective session and had felt that problems were often being saved up for a retrospective even if they could have been addressed earlier.

Fabio previously wrote about the idea of the http://fabiopereira.me/blog/2009/11/15/future-retro-box/[future retrospective box] to ensure that we don't forget anything that happened early on in an iteration when we come to a retrospective but that still doesn't address the problem of only addressing problems in the retrospective!

It seems like a combination of having a pre scheduled retrospective with problem solving on a day to day basis is what we need so I'd be interested to know how others have addressed this.
Git/Mercurial: Pushing regularly
Date: 2010-06-19 22:14:06

I was reading a recent blog post by Gabriel Schenker where he discusses http://feedproxy.google.com/~r/LosTechies/~3/h-tL8ABnNkY/git-and-our-friction-points-and-beginners-mistakes.aspx[how his team is making use of Git] and about half way through he says the following:

____
When using Git as your SCM it is normal to work for quite a while -- maybe for a couple of days -- in a local branch and without ever pushing the changes to the origin. Usually we only push when a feature is done or a defect is completely resolved.
____

We've been using Mercurial on  the project I'm currently working on over the past few months and although it's a similar tool we've been following a different approach.

We've got it setup the same way we would setup Subversion:

image::{{<siteurl>}}/uploads/2010/06/dscm.gif[dscm.gif,675]

We've been trying to push to the central repository as frequently as possible, just as we would if we were using Subversion.

I don't know the Git workflow that well because I haven't used it on a project yet but we've always found that it's beneficial to integrate with code being written by others on the team as frequently as possible.

Not doing this can lead to the problems which Martin Fowler outlines in his post about http://martinfowler.com/bliki/FeatureBranch.html[feature branches].

We've tried to ensure that after every commit the build still passes although we do sometimes have broken versions in the code committed locally because we don't run our full test suite before every local check in.

Even if a feature isn't completed I still think it's valuable to have what we've done so far checked in and it also helps remove the problem with needing to backup local repositories:

____
Since we are going to work locally potentially for days without pushing to the origin (our central repository) we might well loose our work if we have a hard disk crash or our office is flooded. Thus we need some backup strategy.
____

We just need to make sure the central repository is being backed up and then the danger of losing our work is significantly reduced.