markhneedham/blog huggingface dataset:
Shell: Create a comma separated string
Date: 2017-06-23 12:26:49

I recently needed to generate a string with comma separated values, based on iterating a range of numbers.

e.g. we should get the following output where n = 3

[source,bash]
----

foo-0,foo-1,foo-2
----

I only had the shell available to me so I couldn't shell out into Python or Ruby for example. That means it's bash scripting time!

If we want to iterate a range of numbers and print them out on the screen we can write the following code:

[source,bash]
----

n=3
for i in $(seq 0 $(($n > 0? $n-1: 0))); do
  echo "foo-$i"
done

foo-0
foo-1
foo-2
----

Combining them into a string is a bit more tricky, but luckily I found http://ajhaupt.blogspot.co.uk/2010/12/create-comma-separated-string-in-shell.html[a great blog post] by Andreas Haupt which shows what to do. Andreas is solving a more complicated problem than me but these are the bits of code that we need from the post.

[source,bash]
----

n=3
combined=""

for i in $(seq 0 $(($n > 0? $n-1: 0))); do
  token="foo-$i"
  combined="${combined}${combined:+,}$token"
done
echo $combined

foo-0,foo-1,foo-2
----

This won't work if you set +++<cite>+++n<0+++</cite>+++ but that's ok for me! I'll let Andreas explain how it works:

* +++<cite>+++${combined:+,}+++</cite>+++ will return either a comma (if +++<cite>+++combined+++</cite>+++ exists and is set) or nothing at all.
* In the first invocation of the loop +++<cite>+++combined+++</cite>+++ is not yet set and nothing is put out.
* In the next rounds +++<cite>+++combined+++</cite>+++ is set and a comma will be put out.

We can see how it in action by printing out the value of +++<cite>+++$combined+++</cite>+++ after each iteration of the loop:

[source,bash]
----

n=3
combined=""

for i in $(seq 0 $(($n > 0 ? $n-1: 0))); do
  token="foo-$i"
  combined="${combined}${combined:+,}$token"
  echo $combined
done

foo-0
foo-0,foo-1
foo-0,foo-1,foo-2
----

Looks good to me!
Kubernetes: Which node is a pod on?
Date: 2017-06-14 08:49:06

When running Kubernetes on a cloud provider, rather than locally using https://github.com/kubernetes/minikube[minikube], it's useful to know which node a pod is running on.

The normal command to list pods doesn't contain this information:

[source,bash]
----

$ kubectl get pod
NAME           READY     STATUS    RESTARTS   AGE
neo4j-core-0   1/1       Running   0          6m
neo4j-core-1   1/1       Running   0          6m
neo4j-core-2   1/1       Running   0          2m
----

I spent a while searching for a command that I could use before I came across https://tachingchen.com/blog/Kubernetes-Assigning-Pod-to-Nodes/[Ta-Ching Chen's blog post] while looking for something else.

Ta-Ching points out that we just need to add the flag +++<cite>+++-o wide+++</cite>+++ to our original command to get the information we require:

[source,bash]
----

$ kubectl get pod -o wide
NAME           READY     STATUS    RESTARTS   AGE       IP           NODE
neo4j-core-0   1/1       Running   0          6m        10.32.3.6    gke-neo4j-cluster-default-pool-ded394fa-0kpw
neo4j-core-1   1/1       Running   0          6m        10.32.3.7    gke-neo4j-cluster-default-pool-ded394fa-0kpw
neo4j-core-2   1/1       Running   0          2m        10.32.0.10   gke-neo4j-cluster-default-pool-ded394fa-kp68
----

Easy!
Python: Combinations of values on and off
Date: 2017-12-03 17:23:14

In my continued exploration of Kaggle's https://www.kaggle.com/c/spooky-author-identification/leaderboard[Spooky Authors competition], I wanted to run a http://scikit-learn.org/stable/modules/ensemble.html#using-the-votingclassifier-with-gridsearch[GridSearch] turning on and off different classifiers to work out the best combination.

I therefore needed to generate combinations of 1s and 0s enabling different classifiers.

e.g. if we had 3 classifiers we'd generate these combinations

[source,text]
----

0 0 1
0 1 0
1 0 0
1 1 0
1 0 1
0 1 1
1 1 1
----

where\...

* '0 0 1' means: classifier1 is disabled, classifier3 is disabled, classifier3 is enabled
* '0 1 0' means: classifier1 is disabled, classifier3 is enabled, classifier3 is disabled
* '1 1 0' means: classifier1 is enabled, classifier3 is enabled, classifier3 is disabled
* '1 1 1' means: classifier1 is enabled, classifier3 is enabled, classifier3 is enabled

\...and so on. In other words, we need to generate the binary representation for all the values from 1 to 2^number of classifiers^-1.

We can write the following code fragments to calculate a 3 bit representation of different numbers:

[source,python]
----

>>> "{0:0b}".format(1).zfill(3)
'001'
>>> "{0:0b}".format(5).zfill(3)
'101'
>>> "{0:0b}".format(6).zfill(3)
'110'
----

We need an array of 0s and 1s rather than a string, so let's use the +++<cite>+++list+++</cite>+++ function to create our array and then cast each value to an integer:

[source,python]
----

>>> [int(x) for x in list("{0:0b}".format(1).zfill(3))]
[0, 0, 1]
----

Finally we can wrap that code inside a list comprehension:

[source,python]
----

def combinations_on_off(num_classifiers):
    return [[int(x) for x in list("{0:0b}".format(i).zfill(num_classifiers))]
            for i in range(1, 2 ** num_classifiers)]
----

And let's check it works:

[source,python]
----

>>> for combination in combinations_on_off(3):
       print(combination)

[0, 0, 1]
[0, 1, 0]
[0, 1, 1]
[1, 0, 0]
[1, 0, 1]
[1, 1, 0]
[1, 1, 1]
----

what about if we have 4 classifiers?

[source,python]
----

>>> for combination in combinations_on_off(4):
       print(combination)

[0, 0, 0, 1]
[0, 0, 1, 0]
[0, 0, 1, 1]
[0, 1, 0, 0]
[0, 1, 0, 1]
[0, 1, 1, 0]
[0, 1, 1, 1]
[1, 0, 0, 0]
[1, 0, 0, 1]
[1, 0, 1, 0]
[1, 0, 1, 1]
[1, 1, 0, 0]
[1, 1, 0, 1]
[1, 1, 1, 0]
[1, 1, 1, 1]
----

Perfect! We can now use this function to help work out which combinations of classifiers are needed.
Morning Pages: What should I write about?
Date: 2017-12-27 23:28:35

I've been http://juliacameronlive.com/basic-tools/morning-pages/[journalling] for almost 2 years now but some days I get stuck and can't think of anything to write about.

I did a bit of searching to see if anybody had advice on solving this problem and found a few different articles:

* https://blog.doist.com/benefits-of-journaling-81b663a27608[The Productive Benefits of Journaling (plus 11 ideas for making the habit stick)]
* https://medium.com/darius-foroux/read-this-if-you-want-to-keep-a-journal-but-dont-know-how-a3f081d360aa[Read This If You Want To Keep A Journal But Don't Know How]
* http://tinybuddha.com/blog/turn-pain-to-joy-11-tips-for-a-powerful-gratitude-journal/[Turn Pain to Joy: 11 Tips for a Powerful Gratitude Journal]
* https://greatergood.berkeley.edu/article/item/tips_for_keeping_a_gratitude_journal[Tips for Keeping a Gratitude Journal]
* https://www.buzzfeed.com/rachelwmiller/how-to-start-a-bullet-journal?utm_term=.vwdGL53Dp#.swwEzg5kD[WTF Is A Bullet Journal And Why Should You Start One? An Explainer]

The articles talk about different approaches to journalling and since I'm not following one particular approach I thought I'd summarise all their ideas and put them in a document that I can use. I've also added some of my own ones.

Here's the list:

* Something you're grateful for
* An unusual event
* Goals or hopes
* A past event
* Things that are stopping you achieving your goals
* Values that are important to you
* Ideas or nagging thoughts
* What did you do yesterday?
 ** What did you work on?
 ** Who did you talk to?
 ** What book did you read?
 ** What podcasts did you listen to?
 ** What TV shows/movies did you watch?
 ** Where did you go?
* What are you doing today?
* What scares you?
* Decisions you need to make/made - small or big

If you have any other ideas of what I can write about let me know in the comments. While researching for this post I noticed that Julia Cameron, the inventor of Morning Pages, has a new book out - https://www.amazon.co.uk/gp/product/B073QXSCWC/ref=oh_aui_d_detailpage_o00_?ie=UTF8&psc=1[The Right to Write: An Invitation and Initiation into the Writing Life] - so I'm hoping to get some ideas from there as well.
Neo4j: Cypher - Property values can only be of primitive types or arrays thereof.
Date: 2017-12-01 22:09:17

I ran into an interesting https://neo4j.com/developer/cypher-query-language/[Cypher] error message earlier this week while trying to create an array property on a node which I thought I'd share.

This was the Cypher query I wrote:

[source,cypher]
----

CREATE (:Person {id: [1, "mark", 2.0]})
----

which results in this error:

[source,cypher]
----

Neo.ClientError.Statement.TypeError
Property values can only be of primitive types or arrays thereof.
----

We actually are storing an array of primitives but we have a mix of different types which isn't allowed. Let's try coercing all the values to strings:

[source,cypher]
----

CREATE (:Person {id: [value in [1, "mark", 2.0] | toString(value)]})

Added 1 label, created 1 node, set 1 property, completed after 4 ms.
----

Success!
Python: Learning about defaultdict's handling of missing keys
Date: 2017-12-01 15:26:36

While reading the scikit-learn code I came across a bit of code that I didn't understand for a while but in retrospect is quite neat.

This is the code snippet that intrigued me:

[source,python]
----

vocabulary = defaultdict()
vocabulary.default_factory = vocabulary.__len__
----

Let's quickly see how it works by adapting an example from scikit-learn:

[source,python]
----

>>> from collections import defaultdict
>>> vocabulary = defaultdict()
>>> vocabulary.default_factory = vocabulary.__len__

>>> vocabulary["foo"]
0
>>> vocabulary.items()
dict_items([('foo', 0)])

>>> vocabulary["bar"]
1
>>> vocabulary.items()
dict_items([('foo', 0), ('bar', 1)])
----

What seems to happen is that when we try to find a key that doesn't exist in the dictionary an entry gets created with a value equal to the number of items in the dictionary.

Let's check if that assumption is correct by explicitly adding a key and then trying to find one that doesn't exist:

[source,python]
----

>>> vocabulary["baz"] = "Mark
>>> vocabulary["baz"]
'Mark'
>>> vocabulary["python"]
3
----

Now let's see what the dictionary contains:

[source,python]
----

>>> vocabulary.items()
dict_items([('foo', 0), ('bar', 1), ('baz', 'Mark'), ('python', 3)])
----

All makes sense so far. If we look at https://github.com/python/cpython/blob/master/Modules/_collectionsmodule.c#L1973[the source code] we can see that this is exactly what's going on:

[source,python]
----

"""
__missing__(key) # Called by __getitem__ for missing key; pseudo-code:
  if self.default_factory is None: raise KeyError((key,))
  self[key] = value = self.default_factory()
  return value
"""
pass
----

scikit-learn uses this code to store a mapping of features to their column position in a matrix, which is a perfect use case.

All in all, very neat!
Riskiest thing first vs Outside in development
Date: 2010-03-02 22:49:11

I had an interesting conversation with my colleague http://ilovemartinfowler.com/[David Santoro] last week where I described the way that I often pick out the http://www.markhneedham.com/blog/2009/05/11/tackling-the-risk-early-on-at-a-task-level/[riskiest parts of a story or task] and do those first and David pointed out that this approach didn't seem to fit in with the idea of http://www.infoq.com/presentations/bdd-dan-north[outside in development].

The idea with outside in development as I understand it is that we would look to drive any new functionality from the UI i.e. the outside and work our way inwards through the various layers and probably eventually end up with persistence i.e. the inside.

In the particular example that were basing our discussion on I described a story that I was working on with my pair where we needed to apply some constraints to certain items in our data set and then display them differently on the UI as a result of that.

We went through the existing domain model to see if there was anything in there that we could make use of, and having realised that there wasn't anything we mapped out the tasks we would need to do to implement this functionality.

The most difficult/tricky task was to handle the data migration as we realised that we would need to add an extra column to one of the tables. We also needed to do the associated Hibernate mapping and the code from the UI and controller level downwards to make use of this.

We did the data migration and associated work first and one we were happy that was working we went and coded from the UI downwards until we reached the persistence layer.

In this situation it seemed like it worked out reasonably well and I couldn't see that we would have ended up with a different solution if we had started off driving through the UI.

In fact we did end up spending most of our time doing the data migration so to me it seemed somewhat justified doing that first since we did run into a couple of problems.

Outside in development in general seems a good thing to me so I'm curious as to whether I'm justifying a sub optimal approach to myself or whether there are some situations where we can vary the approach a bit?
Parallel Pair Programming
Date: 2010-03-16 23:56:47

I've spent a bit of time working with http://oldtinroof.com/[Les] recently and it's been quite interesting working out the best way for us to pair together as he's working as a front end developer on the team which means he's best utilised working on the CSS/JavaScript/HTML side of things.

Having said that there are often features which require both front end and backend collaboration and we've been trying to drive these features from the front end through to the backend rather than working on the backend code separately and then working with Les later on to hook it all up to the frontend.

As a result we've found that it seems to be most effective to prototype any backend code that we write while working together such that we just write enough code to allow us to see that everything is hooked up correctly.

Once we're happy that we've got that working correctly then we split up and work side by side, effectively http://vivekvaid.blogspot.com/2008/09/death-of-paired-programming-its-2008.html[parallel pairing as Vivek Vaid describes it].

At this stage I would focus on driving out the backend code properly with a more test driven approach and Les works on tidying things up from a front end perspective and ensuring that the feature's look and feel is consistent across browsers and doing any other front end related functionality.

The benefit that we've seen from doing this is that we're able to work together on the code where we get the most value from doing so and then split up to work on the other things where we wouldn't add as much value to each other in a pair programming situation.

Since we're working directly together at the beginning of a story we also get the benefit of iterating/talking through the potential approaches that we can take and I've often found that I take a better approach to solving a problem when working with a pair than when working alone.

The thing that we have to be careful about when doing this is ensuring that we're not treading on each other's toes because we are working around the same part of the code base and often on the same files.

This means that there is a bit of merging to do but as we're also sitting next to each other it hasn't proved to be too much of a problem. I think using a source control tool like Git or Mercurial with their merging capabilities would probably make this a non issue.

It seems to me that we're still getting the http://www.markhneedham.com/blog/2008/09/28/pair-programming-what-do-we-gain-from-it/[benefits of pair programming] with this approach and I'm more convinced that this might be a useful approach in other situations as well.
TDD: Expressive test names
Date: 2010-03-19 18:06:51

Towards the end of http://www.markhneedham.com/blog/2009/01/30/tdd-test-dryness/[a post I wrote just over a year ago] I suggested that I wasn't really bothered about test names anymore because I could learn what I wanted from reading the test body.

Recently, however, I've come across several tests that I wrote previously which were testing the wrong thing and had such generic test names that it wasn't obvious that it was happening.

The tests in question were around code which partially clones an object but doesn't copy some fields for various reasons.

Instead of documenting these reasons I had written tests with names like this:

[source,csharp]
----

[Test]
public void ShouldCloneFooCorrectly() { }
----

[source,csharp]
----

[Test]
public void ShouldSetupFooCorrectly() { }
----

When we realised that the code wasn't working correctly, which didn't happen until QA testing, these test names were really useless because they didn't express the intent of what we were trying to test at all.

http://foldingair.blogspot.com/[Damian] and i spent some time writing more fine grained tests which described *why* the code was written the way it was.

We also changed the name of the test fixture to be more descriptive as well:

[source,csharp]
----

[TestFixture]
public class WhenCloningFooTests
{
	[Test]
	public void ShouldNotCloneIdBecauseWeWantANewOneAssignedInTheDatabase() { }

	[Test]
	public void ShouldNotCopyCompletedFlagBecauseWeWantTheFooCompletionJobToPickItUp() { 	
}
----

It seems to me that these new tests names are more useful as http://blog.orfjackal.net/2010/02/three-styles-of-naming-tests.html[specifications of the system behaviour] although we didn't go as far as you can with some frameworks where you can create base classes and separate methods to describe the different parts of a test.

Despite that I think naming tests in this way can be quite useful so I'm going to try and write more of my tests like this.

Of course it's still possible to test the wrong thing even if you are using more expressive names but I think it will make it less likely.
Finding the assumptions in stories
Date: 2010-03-26 01:14:15

My colleague J.K. has written http://jkwerner2.wordpress.com/2010/03/23/so-that-so-what/[an interesting blog post where he describes a slightly different approach that he's been taking to writing stories] to help move the business value in a story towards the beginning of the description and avoid detailing a solution in the 'I want' section of the story.

To summarise, J.K.'s current approach involves moving from the traditional story format of:

[source,text]
----

As I...
I want..
So that...
----

To the following:

[source,text]
----

As I...
I want..
By...
----

I quite like this idea and I've noticed that even without using this story format technical solutions are sometimes described as the business requirement and we need to look beyond the 'I want' section of the story card to find the real value and locate assumptions which have led to the story being written in that way.

To give a recent example, a colleague and I picked up the following story:

[source,text]
----

As the business
I want a HTTP module to be included on the old site
So that I can redirect a small percentage of traffic to the new site
----

We assumed that other options for redirecting traffic must have already been analysed and written off in order for this to be the suggested solution so we initially started looking at how to implement it.

After a bit of investigation it became clear that this was going to be quite an invasive solution to the problem and would involve re-testing of the whole old site (since we would be making a change there) before it could be put into production. That would take 2 weeks.

Speaking with http://www.the-arm.com/[Toni] and http://twitter.com/a5hok[Ashok] about the problem it became clear that it should be possible to  control whether traffic was going to the old or new site by changing the configuration in our load balancer, http://www.citrix.com/English/ps2/products/product.asp?contentID=21679[Netscaler].

Discussing this further we found out that this had been tried previously and hadn't quite worked out as expected which was why it hadn't been considered as an option.

We spent some time talking through using Netscaler with the network team and agreed to try it out on a performance environment and see whether it would balance traffic in the way that we wanted which it did.

We still need to make sure that it works as expected in production but it was an interesting example of how solutions can be excluded based on prior experience even though they might still be useful to us.

I'll certainly be more aware of noticing when a story details a solution and try and look for the actual requirement after this experience.
Saved from an episode of bear shaving
Date: 2010-03-30 06:57:43

As part of our continuous integration build we have a step in the build which tears down a Windows service, uninstalls it and then reinstalls it later on from the latest files checked into the repository.

One problem we've been having recently is that despite the fact it should already have been uninstalled a lock has been kept on the log4net dll in our build directory, a directory that we tear down as one of the next steps.

It was a bit of a sporadic failure and whenever we went on the build machine and checked for file handles with http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx[Process Explorer] the log4net dll never showed up on the list.

I wrongly assumed that it couldn't be getting locked by the Windows service because the Windows service would already be gone by that stage.

We therefore started on a http://www.markhneedham.com/blog/2009/08/06/bear-shaving/[bear shaving] expedition which involved trying to make use of http://www.emptyloop.com/unlocker/[Unlocker] and http://download.cnet.com/FileAssassin/3000-2094_4-10639988.html[File Assassin] to release the file handle so that we'd be able to delete the directory.

We weren't able to get either of those tools to do the job from the command line and luckily at this stage another colleague had the idea to check whether or not the Windows service was actually immediately removed in Task Manager when we tore it down.

After realising that it wasn't we were able to address this problem and put a step in the build to look for that process and kill it as part of the build.

I guess the real solution to this problem would be for the uninstall to block until it was properly uninstalled but this seems to be the closest we can get to addressing the cause rather than the effect.
A reminder about context switching
Date: 2010-03-01 23:12:01

I've spent most of my time working on agile software development teams over the last few years so for the most part each pair is only working on one story, keeping the work in progress low and allowing them to focus on that piece of work until it's completed.

My pair and I ended up in a therefore somewhat unusual situation last week where we were attempting to work on three things at the same time and weren't doing a particularly great job on any of them.

It wasn't immediately obvious to me that we were doing this since the two extra tasks that we were working on were related to deployment issues on different environments.

However we eventually started making mistakes and in rushing to rectify those made even more mistakes since we were still trying to concentrate on three different things.

It became much more obvious at this point that we needed to just pick one of the items and focus on that until we were done which also served as a reminder that it's good to use the story board as an indicator of what everyone is working on.

In this case two of the tasks we were working on weren't on the story board otherwise it would have been more obvious to the rest of the team that we shouldn't have been working on two of them.

I've never really noticed the problems of context switching before so it was interesting to get such a stark example to remind me of its dangers.