markhneedham/blog huggingface dataset:
Theory of Constraints: Blaming the bottleneck
Date: 2010-12-26 00:04:54

I've been reading http://www.amazon.com/Goal-Process-Ongoing-Improvement/dp/0884271781/ref=sr_1_1?ie=UTF8&qid=1293318612&sr=8-1[The Goal] over the last week or so where Eliyahu Goldratt describes the http://en.wikipedia.org/wiki/Theory_of_Constraints[theory of constraints] as a philosophy for allowing organisations to continually achieve their goal.

Goldratt goes on to describe http://en.wikipedia.org/wiki/Bottleneck[bottlenecks] - resources which have a capacity less than the capacity being demanded of the system.

The capacity of the system cannot be higher than that of the bottleneck which means that we need to find a way to optimise the bottlenecks in any system.

In a software context I think an agile/lean approach to delivery is quite effective for allowing us to quickly see where the bottlenecks are - this will typically be where the cards are piling up on the wall.

The problem with this is that we can frequently end up blaming the people working in the role where the bottleneck seems to be.

On the projects I've worked on stories tend to get stuck in QA since there are significantly less testers than developers on any team.

Since blaming isn't particularly useful Goldratt suggests two ways to optimise bottlenecks:

. Don't waste time of the bottleneck
 ** Processing defective parts
 ** Sitting idle
 ** Work on unnecessary parts
. Take load off the bottleneck

The bottleneck ends up sitting idle when we wait until we've completely finished a story before showcasing what we've done to the testers.

It's frequently the case that we can QA test a piece of functionality before the whole thing is complete.

We can avoid processing defective parts in this case by talking through a story with a tester before we start and thinking of the bugs we might create in development. Unit testing can also be useful for reducing defects.

I'm not sure how working on unnecessary parts would work most of the time - perhaps testing features which aren't required for the next release would be an example of this.

We can achieve the second to an extent by ensuring that we have more scenarios automated so that a tester doesn't need to manually cover them each time.

We could also get developers to temporarily take on the role of a tester to increase capacity but from my experience that doesn't seem to work out as well in practice as it seems like it should.
Ruby: One method, two parameter types
Date: 2010-12-07 05:01:44

One interesting thing that I've noticed while coding in Ruby is that due to the dynamicness of the language it's possible to pass values of different types into a given method as parameters.

For example, I've recently come across a few examples of methods designed like this:

[source,ruby]
----

def calculate_foo_prices(foos)
   ...
   [foos].flatten.each do |foo|
      # do something
   end
end
----

This allows us to use the method like this:

[source,ruby]
----

# foos would come in as an array from the UI
foos = [Foo.new, Foo.new, Foo.new]
calculate_foo_prices(foos)
----

Or like this:

[source,ruby]
----

calculate_foo_prices(Foo.new)
----

It becomes quite confusing to understand why what is supposedly already a collection is being put inside an array on line 3 of the first example when you first read it.

An alternative would be to pull out a different method for calculating the price of the single Foo:

[source,ruby]
----

def calculate_foo_price(foo)
   calculate_foo_prices([foo])
end
----

And then simplify the original method:

[source,ruby]
----

def calculate_foo_prices(foos)
   ...
   foos.each do |foo|
      # do something
   end
end
----

While writing this I was thinking that another way could be to change the original method to look like this by using the splat operator:

[source,ruby]
----

def calculate_foo_prices(*foos)
   ...
   foos.each do |foo|
      # do something
   end
end
----

Which means that we can use the same method for both situations:

[source,ruby]
----

calculate_foo_prices(Foo.new)
----

[source,ruby]
----

# foos would come in as an array from the UI
foos = [Foo.new, Foo.new, Foo.new]
calculate_foo_prices(*foos)
----

I'm guessing the latter is more idiomatic Ruby or perhaps there's another way I'm not aware of yet?
Ruby: Exiting a 'loop' early
Date: 2010-12-01 17:56:51

We recently had a problem to solve which at its core required us to iterate through a collection, look up a value for each key and then exit as soon as we'd found a value.

The original solution looped through the collection and then explicitly returned once a value had been found:

[source,ruby]
----

def iterative_version
  v = nil
  [1,2,3,4,5].each do |i|
    v = long_running_method i
    return v unless v.nil?
  end
  v
end
----

[source,ruby]
----

def long_running_method(value)
  puts "inside the long running method with #{value}"
  return nil if value > 3
  value
end
----

Which we run like so:

[source,ruby]
----

p "iterative value is #{iterative_version.to_s}"
----

This prints the following when we run it:

[source,text]
----

inside the long running method with 1
"iterative value is 1"
----

I figured there must be a more functional way to solve the problem and I eventually came up with this:

[source,ruby]
----

def functional_version
  [1,2,3,4,5].map {|i| long_running_method i }.find { |i| !i.nil? }
end
----

Which prints the following when we run it:

[source,ruby]
----

inside the long running method with 1
inside the long running method with 2
inside the long running method with 3
inside the long running method with 4
inside the long running method with 5
"functional value is 1"
----

The problem is that collections in Ruby are eager evaluated so we evaluate every single item in the collection before we get the first non nil value.

Luckily the http://flori.github.com/lazylist/[lazylist] gem comes to our rescue and allows us to solve the problem in a functional way:

[source,ruby]
----

require 'lazylist'
def lazy_version
  lazy_list([1,2,3,4,5]).find { |i| !i.nil? }
end

def lazy_list(values)
  list(long_running_method(values.first)) { lazy_list(values - [values.first]) }
end
----

Running that gives us this:

[source,text]
----

inside the long running method with 1
"lazy value is 1"
----

I've never come across a problem where I needed to use a lazy list but finally I have and I think the version which uses it is pretty neat.
India Cultural Differences: Language
Date: 2010-12-24 18:12:51

For the majority of the time that I've spent in Pune so far language hasn't been a big deal at all but there are a couple of differences that I didn't initially anticipate.

== The local language

While the official office language is English my colleagues seem more comfortable talking to each other in Hindi so quite frequently the conversation will move into Hindi if someone isn't directly speaking to me.

I initially found it tremendously frustrating that I couldn't understand what was going on in group discussions and while I did point it out people don't realise when they're switching into Hindi so it's a difficult problem to fix.

My brain now seems to zone out when people stop speaking in English so I don't even notice that it's happened unless I observe myself becoming much more aware of my surroundings because I've tuned out and started looking around the room.

An interesting side effect is that I don't pick up as much information from http://alistair.cockburn.us/Osmotic+communication[osmotic communication] as I have done when working in the UK/Australia because conversations between my colleagues have a 50/50 chance of being in Hindi.

I guess the easiest way for me to fix the problem would be to take lessons in Hindi but I've never got around to that.

Another interesting thing to note is that it's not only me who doesn't understand Hindi but also people from the Chennai office.

== Swearing

Swearing in India is a bit of a taboo which is quite strange for me because in the UK/Australia there's not really any stigma attached to the majority of words.

There are a few people who are more easily offended so you have to be a bit more careful about word selection when speaking with them.

Having said that, with the majority of people that I've worked with the language that I use makes no difference at all.
India Cultural Differences: Tolerance/Patience
Date: 2010-12-15 19:08:56

Some colleagues have been asking me recently what cultural differences I've noticed working in India compared to my experiences in the UK and Australia and one of the biggest differences by far is the amount of tolerance and patience people here have compared to me.

These attributes seem to show themselves in roughly two situations:

== With respect to the environment

We've had some building work done in the Pune office recently which has meant that there's been extremely high volume drilling being done to the extent that you can barely hear someone who's sitting a couple of metres away.

I was quite amazed to watch colleagues continuing with their conversations as if nothing had happened - I was totally distracted and had to go and sit in a meeting room away from the noise.

I've talked to a few colleagues about this and they pointed out that they're pretty used to the environment not being perfect from previous experiences at school or just in general life situations so it's not such a big deal.

I find it interesting to see how influential the environment (as far as I can see) can be on some fairly fundamental personality traits.

== With people

People here seem to give others the benefit of the doubt much more frequently than I've seen elsewhere.

I've noticed this especially when communicating with people in remote locations.

I worked on a project distributed between two cities a couple of years ago and the distrust between the two locations was fairly evident in all methods of communication.

I haven't noticed the same thing here as much and the majority of the time people don't react at all to something which I would consider to be inflammatory.

There's also much more tolerance for repetition of information than I've noticed elsewhere.

I get very frustrated if I hear the same thing repeated multiple times but the majority of people here don't seem too bothered by that and take it in their stride.

== It's a strength not a weakness

I think that people here sometimes don't recognise that their tolerance/patience is a strength rather than a weakness and I've frequently heard conversations where what I consider being tolerant is seen to be 'rolling over'.

I don't necessarily think the 'western' style of presenting an argument very assertively/confidently is the only way to go and I've seen colleagues successfully communicate while using their own style.

Of course these are massive generalisations based on my observations so it'd be interesting to see what other people who've had the chance to work in both cultures/environments think.
Distributed Agile: Other observations
Date: 2010-12-12 08:11:31

Some of the difficulties of working in an offshore environment were clear to me before I even came to India but I've come across a few others lately which I either didn't think about before or didn't realise how annoying they were!

== Getting data from the client's network

For several of the stories that we've been working on lately we needed to make use of huge amounts of reference data residing on the client's network.

If we were working in the same building then it would be very quick to download that and use it for whatever we needed.

Working offshore we've been downloading those dumps via scp which can take around 5 hours for a 300 MB file.

I don't know a lot about networks but I'm told that it would be possible to create a direct VPN between our office and the client's which would allow the transfer of this type of data to be much quicker.

== Dealing with internal IP addresses

Since we're working on a different network any IP addresses for test environments of external systems aren't directly accessible.

We have a box setup in our office connected via a VPN to the client's network which forwards requests we make to local IP addresses to the equivalent IPs on the client's network.

There have been several occasions when there have been internal IP addresses hard coded in our configuration files and we've completely forgotten that we need to create an equivalent local address.

It's not a big time waster but it's something you wouldn't have to think about if working in the client's office.

== Slowness of the feedback cycle

It's obvious that if the client is in a different country to you that there will be less face to face communication but I didn't realise how frustrating that would be.

If you have an observation about a piece of functionality and the offshore BAs don't have the answer then you'll most likely have to wait 5/6 hours to speak to someone onshore about it or write your thoughts in an email.

While both of those are viable options I find my motivation to express my opinion is much lower if I can't get feedback straight away.

As a result of this communication lag I think that we spend more time building functionality which doesn't necessarily add a lot of value.
Bugs: Prioritising by bucket
Date: 2010-12-12 07:59:17

At a lot of organisations that I've worked there is a tendency to prioritise bugs by a priority bucket.

We might therefore have priority buckets 1-4 where the bucket number indicates how important the bug is to fix and then any buckets ranked below 4 would not be fixed but would be logged anyway.

From what I've noticed this isn't a particularly effective way of managing bugs.

To start with there tend to be a lot of discussions around what the priority of each bug should be where a QA will argue that it should be a higher priority while a developer disagrees.

I'm not convinced that we actually get any value out of these discussions.

Another thing that I've noticed recently is that developers are much less motivated to fix bugs if they've been put in Bucket 4 rather than one of the higher priority buckets.

I think a better way to solve this problem is just to *store whether or not we're going to fix the bug before the next release or not*. We can then list them in the order that they should be fixed as well.

It's a much simpler binary categorisation and there's no stigma attached to picking up a 'low priority' bug because each bug picked up is just the next one on the list.

We did this on one project I worked on and it was interesting to notice the number of bugs the client was completely disinterested in fixing before the next release because they just weren't that important.

I'm not convinced the same thing would have happened if we'd categorised by bucket.

The added benefit of this approach is that we can avoid discussions where we identify how many bugs of each category have been identified each iteration.

In some organisations the ability to detect higher priority bugs is an indication of the quality of QA work but I don't believe this is a useful metric to track.
Technical implementation heavy stories
Date: 2010-12-13 21:29:02

Earlier this year I wrote about http://www.markhneedham.com/blog/2010/03/26/finding-the-assumptions-in-stories/[some of the problems that we can run into when we have implicit assumptions in stories] and another problematic approach I've seen around this area is where we end up with stories that are very heavily focused on technical implementation.

Initially this seems like it will work out pretty well since all the developer then needs to do is follow the steps that have been outlined for them but from my experience it seems to create more problems than it solves.

The first problem is that it seems to create a *non thinking mindset* and the more detail there is the less it feels like you need to think.

This isn't a great position to end up in since there are frequently multiple ways to solve any problem and we will probably exclude those from our thinking if we're just following instructions.

As a developer part of the skill-set is knowing how to implement a solution but an equally important part is being able to *suggest alternative approaches* which may be more appropriate in a given context.

Technically heavy stories only encourage the former as the developer may now assume that any other options have somehow already been considered.

In addition to this it tends to take a lot of time to explain these types of stories which presumably also means that it takes longer to write them up as well.

I've also found that it's more difficult to understand these types of stories since it's not initially obvious why you need to implement the technical approach that has been described.

Another thing which I hadn't appreciated until quite recently is that it's very boring/demotivating to implement stories if all the problems have already been 'solved' before you even start.

Given that it's somewhat inevitable that we may end up with a story that is shaped like this the key seems to be not to get frustrated by the story but rather to try and find out what the underlying intent of the requirement actually is.
Ask someone vs work it out yourself
Date: 2010-12-14 18:04:18

Back in 2007/2008 when I worked on my first couple of projects at ThoughtWorks I always found it strange how frequently my colleagues would try and figure something out themselves rather than asking someone else (who already knew how to do it) how to do it.

Fast forward to 2010 and I find myself being the one encouraging people to figure things out themselves.

There's still merit in communicating with colleagues when we've tried to work out how to do something and haven't managed to figure it out but it's also useful to not have this as our default mode.

This seems to be particularly the case when working offshore if we have a reasonable idea that someone onshore might be able to help us with something we're working on.

We can stop and wait for them to start their work day but frequently we'll be able to make some progress without their input and then if we really do need help we can always get it later on.

While someone else can often solve a problem quicker than us the disadvantage is that we don't tend to learn by watching others but rather by struggling ourselves.

In my first job it was suggested to me by a colleague that I should try and solve any problems alone for an hour before asking someone else for help.

Having said that I do think it's still useful to know which people in a team have expertise in certain areas so that we can ask them for help when required.
Communication when it's not going your way
Date: 2010-12-22 23:32:46

I've been reading some of the articles written about the disruption caused by the snow across Europe and I found one quote in http://www.telegraph.co.uk/topics/weather/8214205/UK-snow-flights-cancelled-and-roads-closed-as-Arctic-weather-tightens-grip.html[The Daily Telegraph] by Phillip Hammond particularly interesting

____
"I think whilst people are obviously deeply upset about the inconvenience, particularly at this time of year, of having their travel plans disrupted, most of *what I am hearing is a sense of outrage about the way they were then treated when they were stranded at Heathrow airport.*"
____

I was stuck in Brussels Airport for almost a day and the communication by just about everybody there was as non existent as it sounds like it's been at Heathrow.

I think part of the reason for that was that the people in charge didn't know what was happening and part of it was because they didn't want to give people bad news.

I've noticed the same type of thing happen in organisations when there there's something bad to be communicated or an unpopular decision has been made and needs to be explained.

Often in this situation there will be no further communication because *it's assumed that people won't react well to that communication*.

I don't think that's actually an accurate assumption and in addition not communicating is actually quite a dangerous thing to do because it then puts people in the position that they will now guess why certain things are being done.

More often than not those guesses will be more damning of people in leadership positions than they deserve.

Whenever I've seen someone in a leadership position explain what's actually going on (and the thinking behind it) the response of the people receiving the message has always been much more reasonable than they expect it to be.

I think the people in charge of communicating what was going on in the airports would have had similar results if they'd only communicated something!
Finding famous MPs based on their Wikipedia Page Views
Date: 2019-04-01 05:03:00
Learn how to find out the number of pageviews Wikipedia pages have.

As part of the https://towardsdatascience.com/graphing-brexit-bbe4314cf70[Graphing^] https://medium.com/neo4j/graphing-brexit-clustering-edition-3b966694e723[Brexit^] series of blog posts, I wanted to work out who were the most important Members of the UK parliament, and after a bit of Googling I realised that views of their Wikipedia pages would do the trick.

I initially found my way to https://tools.wmflabs.org/pageviews/?project=en.wikipedia.org&platform=all-access&agent=spider&start=2018-01&end=2019-02&pages=Theresa_May[tools.wmflabs.org^], which is great for exploring the popularity of an individual MP, but not so good if you want to extract the data for 600 of them.

I then came to learn that Wikimedia have https://wikimedia.org/api/rest_v1/[a REST API^] and, https://blog.wikimedia.org/2015/12/14/pageview-data-easily-accessible/[hidden at the bottom of a blog post^] from 2015, a Python library called myviews.
Yay!

It's really easy to use as well.
Installation is via PyPi:

[source, bash]
----
pip install mwviews
----

And then if we want to find the page views from the last week for our current Prime Minister, Theresa May, we can write the following code:

[source, python]
----
from mwviews.api import PageviewsClient

p = PageviewsClient("mark-needham")
views = p.article_views("en.wikipedia", ["Theresa May"],  start="20190324", end="20190331")
----

And now let's iterate over `views` to find the number of pageviews per day:

[source, python]
----
for day in views:
    print(day, views[day])

2019-03-24 00:00:00 {'Theresa_May': 23461}
2019-03-25 00:00:00 {'Theresa_May': 22272}
2019-03-26 00:00:00 {'Theresa_May': 18661}
2019-03-27 00:00:00 {'Theresa_May': 42541}
2019-03-28 00:00:00 {'Theresa_May': 34310}
2019-03-29 00:00:00 {'Theresa_May': 34514}
2019-03-30 00:00:00 {'Theresa_May': 20604}
2019-03-31 00:00:00 {'Theresa_May': 18137}
----

We can extend our example to compute pageviews for multiple people by adding their names to the array, and we'll also extend the date range back to the EU referendum of 2016:

[source, python]
----
people = [
    "Boris Johnson", "Theresa May", "Jacob Rees-Mogg", "Jeremy Corbyn"
]

views = p.article_views("en.wikipedia", people,  start="20160624", end="20190331")
----

That's a lot of days so, rather than printing out each day on its own, let's sum up the pageviews:

[source, python]
----
votes = {person: 0 for person in people }

for key in views.keys():
  for person_key in views[key].keys():
    person = person_key.replace("_", " ")
    if views[key][person_key]:
        votes[person] += views[key][person_key]
----

And who's the most famous of them all?

[source,python]
----
max_width = max([len(key) for key in votes.keys()])
for person in votes:
    print(f"{person:<{max_width}} {votes[person]:,}")

Boris Johnson   5,727,213
Theresa May     12,844,215
Jacob Rees-Mogg 3,631,652
Jeremy Corbyn   5,965,669
----