markhneedham/blog huggingface dataset:
Feedback: Reacting immediately
Date: 2013-05-23 22:43:53

I was recently reading http://www.telegraph.co.uk/sport/football/teams/manchester-united/10066147/Sir-Alex-Fergusons-unparalleled-experience-of-success-and-management-must-not-be-wasted-by-football.htm[an article written by Henry Winter] where he mentioned some of the ideas that Sir ALex Ferguson has been covering in some interviews he's been doing at Harvard and one bit stood out for me:

____
In a series of interviews in Harvard, Ferguson debated dealing with "`fragile`" egos in the dressing room, the power of the two simple words "`well done`" in motivating individuals and *the importance of criticising players`' mistakes immediately after the match and then moving on*.
____

I've written previously about http://www.markhneedham.com/blog/category/feedback/[giving feedback] and I think we don't place enough emphasis on how useful it is to give any feedback close to the time of the event.

My former colleague http://twitter.com/ryangreenhall[Ryan Greenhall] was particularly good at noticing situations where timely feedback was required and I hadn't realised how much of an impact doing that had.

As we as the immediate benefit of the event being fresh in people's minds when they're talking about it we get an added benefit as well.

The more time that we leave between an event happening and reflecting on it, the more time there is for us to invent a story about what happened that helps to explain it.

In the story we create in our mind we'll often associate negative traits/motives to the other person and exaggerate the importance of what happened until it doesn't resemble what actually happened any more.

I tended to veer away from addressing things like this immediately because I thought it seemed like overreacting to believe that it was worth having a conversation but it now seems worthwhile.

In the worst case you just discover there was nothing to reflect on and in the best case you resolve any issues at the source.
R: Mapping over a list of lists
Date: 2013-02-03 10:40:48

As part of the coursera https://class.coursera.org/dataanalysis-001/class/index[Data Analysis] course I had the following code to download and then read in a file:

[source,r]
----

> file <- "https://dl.dropbox.com/u/7710864/data/csv_hid/ss06hid.csv"
> download.file(file, destfile="americancommunity.csv", method="curl")
> acomm <- read.csv("americancommunity.csv")
----

We then had to filter the data based on the values in a couple of columns and work out how many rows were returned in each case:

[source,r]
----

> one <- acomm[acomm$RMS == 4 & !is.na(acomm$RMS)
               & acomm$BDS == 3 & !is.na(acomm$BDS), c("RMS")]
> one
  [1] 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
...
[137] 4 4 4 4 4 4 4 4 4 4 4 4
> two <- acomm[acomm$RMS == 5 & !is.na(acomm$RMS)
               & acomm$BDS == 2 & !is.na(acomm$BDS), c("RMS")]
> two
  [1] 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5
...
[375] 5 5 5 5 5 5 5 5 5 5 5 5

> three <- acomm[acomm$RMS == 7 & !is.na(acomm$RMS)
                 & acomm$BDS == 2 & !is.na(acomm$BDS), c("RMS")]
> three
 [1] 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7
[36] 7 7 7 7 7 7 7 7 7 7 7 7 7 7
----

So I needed to know how many values were in the variables +++<cite>+++one+++</cite>+++, +++<cite>+++two+++</cite>+++ and +++<cite>+++three+++</cite>+++.

I thought I could probably put those lists into another list and then use +++<cite>+++http://www.math.montana.edu/Rweb/Rhelp/apply.html[apply]+++</cite>+++ or one of its variants to get the length of each one.

I usually use the +++<cite>+++http://www.math.montana.edu/Rweb/Rhelp/c.html[c]+++</cite>+++ function to help me create lists but it's not helpful in this case as it creates one massive vector with all the values concatenated together:

Calling apply doesn't have the intended outcome: ~~~r > lapply(c(one, two, three), length) \... [[582]] [1] 1 [[583]] [1] 1 ~~~

Instead what we need is the +++<cite>+++http://www.math.montana.edu/Rweb/Rhelp/list.html[list]+++</cite>+++ function: ~~~r > lapply(list(one, two, three), length) [[1]] [1] 148 [[2]] [1] 386 [[3]] [1] 49 ~~~

Et voila!

The https://github.com/mneedham/dataanalysis2/blob/master/week2.R[code is on github] as usual.
Reading Code: Assume it doesn't work
Date: 2013-02-27 23:12:49

http://www.linkedin.com/profile/view?id=13286237&authType=NAME_SEARCH&authToken=Pfg7&locale=en_US&srchid=f17a4889-19b4-4c0a-b61e-7f84ce7b2cce-0&srchindex=1&srchtotal=1842&goback=%2Efps_PBCK_jae+lee_*1_*1_*1_*1_*1_*1_*2_*1_Y_*1_*1_*1_false_1_R_*1_*51_*1_*51_true_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2_*2&pvs=ps&trk=pp_profile_name_link[Jae] and I have spent a reasonable chunk of the past few weeks pairing on code that neither of us are familiar with and at times we've found it quite difficult to work out exactly what it's supposed to be doing.

My default stance in this situation is to assume that the code is probably correct and then try and work out how that's the case.

After I'd vocalised this a few times, Jae pointed out that we couldn't be sure that the code worked and it didn't make sense to start with that as an assumption.

He pointed out that we were often going down paths of the code that were executed infrequently and since there were no tests around the code we couldn't be sure what it did.

A more useful approach was to *assume that it doesn't work and then reason about it from scratch* to determine whether or not that assumption is correct.

I quite like making this assumption because it makes you concentrate much more closely when you're reading the code rather than skimming over it and assuming that it does what it's supposed to do.

Reading the code has become a mini debugging session where we try to discover how some unfamiliar code actually works.

We've found a couple of bits of code that weren't actually working by following this mindset but it is mentally quite tiring and obviously slower than other styles of reading code!
Regular Expressions: Non greedy matching
Date: 2013-02-16 12:17:49

I was playing around with some football data earlier in the week and I wanted to try and extract just the name 'Rooney' from the following bit of text:

[source,text]
----

Rooney 8′, 27′
----

My initial regular expression was the following which annoyingly captures the time of the first goal:

[source,ruby]
----

> "Rooney 8′, 27′".match(/(.*)\s\d(.*)/)[1]
=> "Rooney 8,"
----

It works fine if the player has only scored one goal...

[source,ruby]
----

> "Rooney 8′".match(/(.*)\s\d(.*)/)[1]
=> "Rooney"
----

\...but since the second part of the regex ("\s\d(.*)") appears twice only the last instance of it is matched and the rest of the text gets captured by the first part of the regex.

One way around this is to make the first part of the regex http://robertpyke.blogspot.co.uk/2009/02/ruby-regex-non-greedy-operator-using_05.html[non greedy/lazy] so that it will make as little as possible rather than as much as possible.

We can do that by using the lazy version of '*' which is '*?':

[source,ruby]
----

> "Rooney 8′, 27′".match(/(.*?)\s\d(.*)/)[1]
=> "Rooney"
----

Of course you could also argue that I'm being very lazy by using ".*" in the first place and you probably have a point! The following more explicit regular expression achieves the same thing:

[source,ruby]
----

> "Rooney 8′, 27′".match(/([A-Za-z\s-]+)\s\d(.*)/)[1]
=> "Rooney"
----

As a side note I find that when I'm playing around with regular expressions it really makes sense to have a bunch of test cases that I can run after each change to make sure I haven't inadvertently broken everything.

http://www.regexper.com/[regexper.com] is also really helpful for visualising what the regular expression is actually doing!
Vertical/Horizontal Slicing
Date: 2013-02-28 22:23:27

A few years ago I wrote a bunch of posts http://www.markhneedham.com/blog/2010/03/02/riskiest-thing-first-vs-outside-in-development/[exploring] http://www.markhneedham.com/blog/2010/04/18/coding-another-outside-in-example/[my experiences] http://www.markhneedham.com/blog/2009/12/19/coding-an-outside-in-observation/[of outside in development] eventually coming to the conclusion that it seemed to make sense to drive out functionality from the UI and work back from there.

i.e. we take a vertical slice of functionality and then drive it end to end.

On the team I'm working on there's been success using an approach where the functionality is still split vertically but we work across a horizontal layer for all the cards before moving onto the next layer.

The advantage of this approach is that *we can do all the work in one layer without context switching* and then move onto the next layer.

In our case this meant adding a bunch of text and check boxes to a backend CMS and then making sure that the values entered there bubbled their way up to a service used by the front end.

This was counter to my previous experience so I was curious why we weren't seeing the problems that I'd seen before where we'd model things incorrectly in the backend and only realise when we tried to call them from the front end.

http://www.linkedin.com/search/fpsearch?fname=Jae&lname=Lee&keepFacets=Y&facet_G=gb%3A0&pplSearchOrigin=TSEO_SN&trk=TSEO_SN[Jae] pointed out that you don't necessarily run into this problem *if you can hold the model of how the whole system fits together in your head*.

Since we have a few people who are able to do that and have knowledge of how the data flows through the different applications that is indeed the case!

I'm sure there are some other cases where this approach makes more sense so if you find success writing code in this style do let me know!
neo4j: Loading data - REST API vs Batch Import
Date: 2013-02-28 23:36:13

A couple of weeks ago when I first started playing around with my football data set I was loading all the data into neo4j using the REST API via https://github.com/maxdemarzi/neography[neography] which was taking around 4 minutes to load.

The data set consisted of just over 250 matches which translated into 8,000 nodes & 30,000 relationships so it's very small by all means.

https://twitter.com/a5hok[Ashok] and I were discussing how that could be quicker and the first thing we tried was to store inserted nodes in an in memory hash map and look them up from there rather than doing an index lookup each time.

These were the timings for different numbers of matches when I did that:

[source,text]
----

--------------------------------------------------------------------
| Matches | Cache-Hits | Cache-Misses | Lucene        | In memory  |
--------------------------------------------------------------------
| 25      | 501        | 325          | 26.692s       | 22.877s    |
| 50      | 1275       | 373          | 50.491s       | 38.304s    |
| 263     | 8016       | 480          | 4m 11.031s    | 2m 49.951s |
--------------------------------------------------------------------
----

For the full data set it was about 30% faster which was a nice improvement but still left me waiting around for a bit longer than I wanted to!

I've http://www.markhneedham.com/blog/2012/09/23/neo4j-the-batch-inserter-and-the-sunk-cost-fallacy/[previously used the batch inserter] and I was planning to use that again to get a significant improvement in loading time until Ashok pointed out Michael Hunger's https://github.com/jexp/batch-import[batch-import] which seemed worth a try.

I had to add an extra step to the http://www.markhneedham.com/blog/2013/02/18/micro-services-style-data-work-flow/[pipeline] to put all the nodes and relationships into CSV files and then pass those files to the batch-import JAR.

There was a massive improvement in the load time using it. These were the timings:

[source,text]
----

-------------------------------------------------------------------
| Matches | Lucene        | In memory  | Batch Import             |
-------------------------------------------------------------------
| 25      | 26.692s       | 22.877s    | 0.378s + 0.921s = 1.299s |
| 50      | 50.491s       | 38.304s    | 0.392s + 1.025s = 1.417s |
| 263     | 4m 11.031s    | 2m 49.951s | 0.524s + 1.239s = 1.763s |
-------------------------------------------------------------------
----

_(the two numbers represent the time taken to generate the CSV files and then the time to import them)_

From my brief skimming of the code it seems to take in the files and then route them through the batch importer API so I imagine similar results would be had by calling that directly.

I know this is not a very fair comparison given that you probably shouldn't be using the REST API to insert data but since I've done it a couple of times I thought it'd be interesting to measure anyway!
Compatible Opinions & Confirmation Bias
Date: 2013-02-28 21:57:11

In 2011 https://twitter.com/thejayfields[Jay Fields] wrote a blog post in which he suggested that it's better to build teams in which people have http://blog.jayfields.com/2011/01/compatible-opinions-on-software.html[a similar opinion on the way software should be built at a high level] rather than having people whose opinions are in conflict.

He referred to this as having 'compatible opinions on software' and since I read the post I've become much more aware of this myself on the teams that I've worked on.

_(I'm sure noticing this is some sort of cognitive bias but I'm http://en.wikipedia.org/wiki/List_of_biases_in_judgment_and_decision_making[not sure which one]!)_

Some examples that Jay gives of incompatible opinions are preference of a powerful language vs powerful IDE or monolithic application vs micro services.

I've noticed some other opinions which seem incompatible as well but what I find most intriguing is that people (including me!) use *extreme confirmation bias to try and show that their opinion is better*.

For example on one team I worked on there was a difference in opinion on whether we should use an ORM to take care of our mapping or if we should write custom SQL to do that job.

We chose to do the latter on that project but whenever we encountered situations where you had to write a lot of SQL it was pointed out how much easier life would be if we had just let the ORM do its job.

On the other hand we had none of the performance problems that you can often run into when you abstract away the database from your application and this fact was frequently pointed out as well!

I think it's probably fair to say that people have some problems they are more interested in solving and others which they don't really care for.

For example someone who's really into Clojure is going to be completely disinterested in trying to solve a problem involving collections using Java when they know that they can solve it much more easily with a few functions.

Equally a Java programmer may lose patience at how long it takes to work out how to use a low level Java API using Clojure when they would have been able to work it out much more quickly using auto complete in a Java IDE.

My general take away is that if I notice myself or others using confirmation bias to prove a point then assuming we both have experience of both approaches it may well be the case that our opinions aren't compatible.

We can then either choose to work on different teams if that's an option or if not then at least acknowledge what's going on with our opinions!

_Brief footnote so I don't get flamed!_

I'm sure there are ways of exploring Java APIs easily using Clojure but hopefully the main point still comes across!
Data Science: Don't filter data prematurely
Date: 2013-02-17 20:02:31

Last year I wrote a post describing how I'd http://www.markhneedham.com/blog/2012/09/30/data-science-scrapping-the-data-together/[gone about getting data for my ThoughtWorks graph] and one mistake about my approach in retrospect is that I filtered the data too early.

My workflow looked like this:

* Scrape internal application using web driver and save useful data to JSON files
* Parse JSON files and load nodes/relationships into neo4j

The problem with the first step is that I was trying to determine up front what data was useful and as a result I ended up running the scrapping application multiple times when I realised I didn't have all the data I wanted.

Since it took a couple of hours to run each time it was tremendously frustrating but it took me a while to realise how flawed my approach was.

For some reason I kept tweaking the scrapper just to get a little bit more data each time!

It wasn't until https://twitter.com/a5hok[Ashok] and I were doing some similar work and had to extract data from an existing database that I realised the filtering didn't need to be done so early in the process.

We weren't sure exactly what data we needed but on this occasion we got everything around the area we were working in and looked at how we could actually use it at a later stage.

Given that it's relatively cheap to store the data I think this approach makes sense more often than not - *we can always delete the data if we realise it's not useful to us at a later stage*.

It especially makes sense if it's difficult to get more data either because it's time consuming or we need someone else to give us access to it and they are time constrained.

If I could rework that work flow it'd now be split into three steps:

* Scrape internal application using web driver and save pages as HTML documents
* Parse HTML documents and save useful data to JSON files
* Parse JSON files and load nodes/relationships into neo4j

I think my experiences tie in reasonably closely with those I heard about at http://www.markhneedham.com/blog/2012/10/02/strata-conf-london-day-1-wrap-up/[Strata Conf London] but of course I may well be wrong so if anyone has other points of view I'd love to hear them.
R: Modelling a conversion rate with a binomial distribution
Date: 2013-02-07 01:26:12

As part of some work https://twitter.com/siddharthdawara[Sid] and I were doing last week we wanted to simulate the conversion rate for an A/B testing we were planning.

We started with the following function which returns the simulated conversion rate for a given conversion rate of 12%:

[source,r]
----

generateConversionRates <- function(sampleSize) {
	sample_a <- rbinom(seq(0, sampleSize), 1, 0.12)
	conversion_a <- length(sample_a[sample_a == 1]) / sampleSize

	sample_b <- rbinom(seq(0, sampleSize), 1, 0.12)
	conversion_b <- length(sample_b[sample_b == 1]) / sampleSize
		
	c(conversion_a, conversion_b)
}
----

If we call it:

[source,r]
----

> generateConversionRates(10000)
[1] 0.1230 0.1207
----

We have a 12.3% conversion rate on A and a 12.07% conversion rate on B based on 10,000 sample values.

We then wrote the following function to come up with 1000 versions of those conversion rates:

[source,r]
----

generateSample <- function(sampleSize) {
	lapply(seq(1, 1000), function(x) generateConversionRates(sampleSize))
}
----

We can call that like this:

[source,r]
----

> getSample(10000)
[[998]]
[1] 0.1179 0.1216

[[999]]
[1] 0.1246 0.1211

[[1000]]
[1] 0.1248 0.1234
----

We were then using these conversion rates to try and work out how many samples we needed to include in an A/B test to have reasonable confidence that it represented the population.

We actually ended up abandoning that exercise but I thought I'd record the code because I thought it was pretty interesting.
Ruby: Stripping out a non breaking space character (&nbsp;)
Date: 2013-02-23 15:04:58

A couple of days ago I was playing with some code to scrape data from a web page and I wanted to skip a row in a table if the row didn't contain any text.

I initially had the following code to do that:

[source,ruby]
----

rows.each do |row|
  next if row.strip.empty?
  # other scraping code
end
----

Unfortunately that approach broke down fairly quickly because empty rows contained a http://www.yellowpipe.com/yis/tools/ASCII-HTML-Characters/index.php[non breaking space] i.e. '&nbsp;'.

If we try called +++<cite>+++strip+++</cite>+++ on a string containing that character we can see that it doesn't get stripped:

[source,ruby]
----

# it's hex representation is A0
> "\u00A0".strip
=> " "
> "\u00A0".strip.empty?
=> false
----

I wanted to see whether I could use +++<cite>+++gsub+++</cite>+++ to solve the problem so I tried the following code which didn't help either:

[source,ruby]
----

> "\u00A0".gsub(/\s*/, "")
=> " "
> "\u00A0".gsub(/\s*/, "").empty?
=> false
----

A bit of googling led me to this http://stackoverflow.com/questions/3913900/ruby-1-9-strip-not-removing-whitespace[Stack Overflow post] which suggests using the http://en.wikipedia.org/wiki/Regular_expression#POSIX_character_classes[POSIX space character class] to match the non breaking space rather than '\s' because that will match more of the different space characters.

e.g.

[source,ruby]
----

> "\u00A0".gsub(/[[:space:]]+/, "")
=> ""
> "\u00A0".gsub(/[[:space:]]+/, "").empty?
=> true
----

So that we don't end up indiscriminately removing all spaces to avoid problems like this where we mash the two names together\...

[source,ruby]
----

> "Mark Needham".gsub(/[[:space:]]+/, "")
=> "MarkNeedham"
----

\...the poster suggested the following regex which does the job:

[source,ruby]
----

> "\u00A0".gsub(/\A[[:space:]]+|[[:space:]]+\z/, '')
=> ""
> ("Mark" + "\u00A0" + "Needham").gsub(/\A[[:space:]]+|[[:space:]]+\z/, '')
=> "Mark Needham"
----

* \A matches the beginning of the string
* \z matches the end of the string

So what this bit of code does is match all the spaces that appear at the beginning or end of the string and then replaces them with ''.
Onboarding: Sketch the landscape
Date: 2013-02-15 07:36:06

For four months during 2012 I was working on the http://digital.cabinetoffice.gov.uk/2012/09/28/meet-the-hosting-and-infrastructure-team/[GDS infrastructure team] and one of the first tasks that https://twitter.com/garethr[Gareth] suggested I do was update a diagram showing how all the different applications and databases worked together.

I thought this was quite a strange thing to ask the 'new guy' to do since I obviously knew nothing at all about how anything worked but he told me that was partly why he wanted me to do it.

I had to go through the puppet and application repositories to work out what monitoring we already had in place, which applications used MongoDB, which used MySQL, which called each other and so on.

If anything didn't make sense I could go and ask someone in one of the application teams and then add my new understanding to the diagram.

As http://2012.33degree.org/talk/show/67[micro services] driven architectures become more and more popular it strikes me that this probably wouldn't be a bad starting point for anyone new to a system.

When you're new you don't yet have the familiarisation that makes things 'obvious' so you're less likely to skip over things and more likely to be inquisitive about why things are the way they are.

As you come across things that don't make sense others who have worked there for longer can play http://www.markhneedham.com/blog/2009/04/25/pimp-my-architecture-dan-north/[the role of the shaman] who explains the history and context behind the decisions which have led to the current state.

Although diagrams can quickly go out of date I still think one of the whole landscape is useful as it gives everyone a common point of reference which can help reduce misunderstandings.

http://www.thekua.com/atwork/[Pat Kua] has http://www.infoq.com/articles/pat-kua-onboarding-new[a bunch of articles covering various different onboarding activities] which are probably worth a read if this is your type of thing.