markhneedham/blog huggingface dataset:
The Language of Risk
Date: 2011-12-30 03:38:58

A few weeks ago Chris Matts wrote an interesting blog post 'http://theitriskmanager.wordpress.com/2011/12/12/the-language-of-risk/[the language of risk]' in which he describes an approach he used to explain the processes his team uses to an auditor.

____
Why did the auditor like what I said? Because *I explained everything we did in terms of risk*. When they asked for a "`process`", I explained the risk the process was meant to address. I then explained how our different process addressed the risk more effectively.
____

This seems like a pretty cool idea to me and it got me thinking of the different 'processes' we've used in teams I've worked on and what risks they might be addressing:

* Pair Programming
 ** Becoming dependent on one person with respect to knowledge of part of the code base.
 ** Having someone new working on an area of the code that they don't know well and making a mistake.
* Retrospective
 ** Making the same mistakes repeatedly/working in a way that (indirectly) wastes money.
* Story Kick Off
 ** Building the wrong thing
 ** Solving the business problem in an inefficient way
 ** Building something which is very difficult to test
* Stand Up
 ** Someone getting stuck on something which someone else in the group might be able to help with.
 ** People going down rabbit holes and getting stuck on things that don't really matter
* Show Case
 ** Building the wrong thing for too long
* Automated testing
 ** The application regresses as new functionality is added
 ** Humans make mistakes when manually going through scenarios

That's just a first attempt at this, I'm sure others could come up with something better!

In coming up with the list I've been working from a process which I've seen used and trying to work out what risk that might be addressing.

Chris seems to look at risks/processes the other way around to i.e. we *think about what risks we need to address and then work out whether we need a process to address it* and if so which one.

Taking that approach would help to explain why some teams don't necessarily need a lot of process - the risks might be catered for in different ways or maybe they just don't exist in specific contexts.

For example a lot of risks around communication go away if the product owner and the team are sitting in the same physical location and can easily just turn and talk to each other if they have any questions.

Even with this new way of looking at risks/process I still think it's useful to keep checking whether or not a process is still necessary because as our team/product changes the risks we face probably do as well.
Learning Android: Deploying application to phone from Mac OS X
Date: 2011-12-23 22:55:17

I've been playing around a little bit today with writing an Android application and while for the majority of the time I've been deploying to an emulator I wanted to see what it'd look like on my phone.

http://developer.android.com/guide/developing/device.html[The developer guide contains all the instructions] on how to do this but unfortunately I'm blessed with the ability to skim over instructions which meant that my phone wasn't getting picked up by the http://developer.android.com/guide/developing/tools/adb.html[Android Debug Bridge].

We can check which devices have been detected by running the following command from wherever the SDK is installed:

[source,text]
----

/Users/mneedham/github/android/android-sdk-macosx/platform-tools/adb devices
----

This was initially returning no devices and http://esausilva.com/2010/10/02/how-to-set-up-adb-android-debug-bridge-in-mac-osx/[reading Esau Silva's blog] made me realise that I'd failed to follow this instruction:

____
On the device, go to Settings > Applications > Development and enable USB debugging (on an Android 4.0 device, the setting is located in Settings > Developer options).
____

I'd assumed it was only necessary if I wanted to debug the application on my phone but it seems like you need to set it to be able to deploy in the first place.

And now my phone gets picked up and can be deployed to, yay!

[source,text]
----

/Users/mneedham/github/android/android-sdk-macosx/platform-tools/adb devices

List of devices attached
3933C40945FA00EC	device
----

If it still doesn't work then http://esausilva.com/2010/10/02/how-to-set-up-adb-android-debug-bridge-in-mac-osx/[Esau Silva has some other ideas] which I haven't yet needed to try.
WebDriver: Getting it to play nicely with Xvfb
Date: 2011-12-15 23:19:31

Another thing we've been doing with WebDriver is having it run with the FirefoxDriver while redirecting the display output into the http://en.wikipedia.org/wiki/Xvfb[Xvfb framebuffer] so that we can run it on our continuous integration agents which don't have a display attached.

The first thing we needed to do was set the environment property 'webdriver.firefox.bin' to our own script which would point the display to Xvfb before starting Firefox:

[source,scala]
----

import java.lang.System._
lazy val firefoxDriver: FirefoxDriver = {
  setProperty("webdriver.firefox.bin", "/our/awesome/starting-firefox.sh")
  new FirefoxDriver()
}
----

Our first version of the script looked like this:

/our/awesome/starting-firefox.sh

[source,text]
----

#!/bin/bash

rm -f ~/.mozilla/firefox/*/.parentlock
rm -rf /var/go/.mozilla


XVFB=`which xVfb`
if [ "$?" -eq 1 ];
then
    echo "Xvfb not found."
    exit 1
fi

$XVFB :99 -ac &


BROWSER=`which firefox`
if [ "$?" -eq 1 ];
then
    echo "Firefox not found."
    exit 1
fi

export DISPLAY=:99
$BROWSER &
----

The mistake we made here was that we started Xvfb in the background which meant that sometimes it hadn't actually started by the time Firefox tried to connect to the display and we ended up with this error message:

[source,text]
----

No Protocol specified
Error cannot open display :99
----

We really wanted to keep Xvfb running regardless of whether the Firefox instances being used by WebDriver were alive or not so we moved the starting of Xvfb out into a separate script which we run as one of the earlier steps in the build.

We also struggled to get the FirefoxDriver to kill itself after each test as calling 'close' or 'quit' on the driver didn't seem to kill off the process.

We eventually resorted to putting a 'pkill firefox' statement at the start of our firefox starting script:

/our/awesome/starting-firefox.sh

[source,text]
----

#!/bin/bash

rm -f ~/.mozilla/firefox/*/.parentlock
rm -rf /var/go/.mozilla

pkill firefox

BROWSER=`which firefox`
if [ "$?" -eq 1 ];
then
    echo "Firefox not found."
    exit 1
fi

export DISPLAY=:99
$BROWSER &
----

It's a bit hacky but it does the job more deterministically than anything else we've tried previously.
WebDriver: Getting it to play nicely with jQuery ColorBox
Date: 2011-12-13 23:31:02

As I http://www.markhneedham.com/blog/2011/12/05/continuous-delivery-removing-manual-scenarios/[mentioned in an earlier post about removing manual test scenarios] we've been trying to automate some parts of our application where a user action leads to a http://jacklmoore.com/colorbox/[jQuery ColorBox] powered overlay appearing.

With this type of feature there tends to be some sort of animation which accompanies the overlay so we have to wait for an element inside the overlay to become visible on the screen before trying to do any assertions on the overlay.

We have a simple method to do that:

[source,scala]
----

def iWaitUntil(waitingFor: => Boolean) {
  for (i <- 1 to 5) {
    if(waitingFor) {
      return
    }
   Thread.sleep(200)
  }
}
----

It can then be called like this in our tests:

[source,scala]
----

def driver: WebDriver = new FirefoxDriver()

iWaitUntil(driver.findElements(By.cssSelector(".i-am .inside-the-colorbox h3")).nonEmpty)
driver.findElement(By.cssSelector(".i-am .inside-the-colorbox h3")).getText should equal("Awesome Title")
----

Annoyingly what we noticed was that this wasn't enough and although the h3 element was coming back as being visible it was then failing the following assertion because 'getText' was returning nothing despite the fact that it clearly had text inside it!

http://twitter.com/#!/uday_rayala[Uday] came up with the neat idea of adding an additional wait clause which would wait until the text was non empty so we now have something like this...

[source,scala]
----

def driver: WebDriver = new FirefoxDriver()

iWaitUntil(driver.findElements(By.cssSelector(".i-am .inside-the-colorbox h3")).nonEmpty)
driver.findElement(By.cssSelector(".i-am .inside-the-colorbox h3")).getText should equal("Awesome Title")
iWaitUntil(driver.findElement(By.cssSelector(".i-am .inside-the-colorbox h3")).getText != "")
----

...which seems to do the job nicely.

An alternative approach would have been to disable the animation of jQuery ColorBox just for our tests but the approach we took was a much quicker win at the time.

We did realise later on that we didn't need to write our own wait method since http://seleniumhq.org/docs/04_webdriver_advanced.html[WebDriver has one built into the API] but I guess they both do similar things so it's not such a big problem.
R: substr - Getting a vector of positions
Date: 2016-04-18 19:49:02

I recently found myself writing an R script to extract parts of a string based on a beginning and end index which is reasonably easy using the +++<cite>+++https://stat.ethz.ch/R-manual/R-devel/library/base/html/substr.html[substr]+++</cite>+++ function:

[source,r]
----

> substr("mark loves graphs", 0, 4)
[1] "mark"
----

But what if we have a vector of start and end positions?

[source,r]
----

> substr("mark loves graphs", c(0, 6), c(4, 10))
[1] "mark"
----

Hmmm that didn't work as I expected! It turns out we actually need to use the +++<cite>+++https://stat.ethz.ch/R-manual/R-devel/library/base/html/substr.html[substring]+++</cite>+++ function instead which wasn't initially obvious to me on reading the documentation:

[source,r]
----

> substring("mark loves graphs", c(0, 6, 12), c(4, 10, 17))
[1] "mark"   "loves"  "graphs"
----

Easy when you know how!
R: tm - Unique words/terms per document
Date: 2016-04-11 05:40:06

I've been doing a bit of text mining over the weekend using the R https://cran.r-project.org/web/packages/tm/index.html[tm package] and I wanted to only count a term once per document which isn't how it works out the box.

For example let's say we're writing a bit of code to calculate the frequency of terms across some documents. We might write the following code:

[source,r]
----

library(tm)
text = c("I am Mark I am Mark", "Neo4j is cool Neo4j is cool")
corpus = VCorpus(VectorSource(text))
tdm = as.matrix(TermDocumentMatrix(corpus, control = list(wordLengths = c(1, Inf))))

> tdm
       Docs
Terms   1 2
  am    2 0
  cool  0 2
  i     2 0
  is    0 2
  mark  2 0
  neo4j 0 2

> rowSums(tdm)
   am  cool     i    is  mark neo4j
    2     2     2     2     2     2
----

We've created a small corpus over a vector which contains two bits of text. On the last line we output a TermDocumentMatrix which shows how frequently each term shows up across the corpus. I had to tweak the default word length of 3 to make sure we could see 'am' and 'cool'.

But we've actually got some duplicate terms in each of our documents so we want to get rid of those and only count unique terms per document.

We can achieve that by mapping over the corpus using the +++<cite>+++tm_map+++</cite>+++ function and then applying a function which returns unique terms. I wrote the following function:

[source,r]
----

uniqueWords = function(d) {
  return(paste(unique(strsplit(d, " ")[[1]]), collapse = ' '))
}
----

We can then apply the function like so:

[source,r]
----

corpus = tm_map(corpus, content_transformer(uniqueWords))
tdm = as.matrix(TermDocumentMatrix(corpus, control = list(wordLengths = c(1, Inf))))

> tdm
       Docs
Terms   1 2
  am    1 0
  cool  0 1
  i     1 0
  is    0 1
  mark  1 0
  neo4j 0 1

> rowSums(tdm)
   am  cool     i    is  mark neo4j
    1     1     1     1     1     1
----

And now each term is only counted once. Success!
Neo4j vs Relational: Refactoring - Add a new field/property
Date: 2016-05-22 09:09:27

A couple of months ago I presented http://info.neo4j.com/intro-to-cypher-for-the-sql-developer-lp.html[a webinar comparing how you'd model and evolve a data model using a Postgres SQL database and Neo4j].

This is what the two data models looked like after the initial data import and before any refactoring/migration had been done:

+++<cite>+++Relational+++</cite>+++

image::{{<siteurl>}}/uploads/2016/05/2016-05-22_09-49-23.png[2016 05 22 09 49 23,400]

+++<cite>+++Graph+++</cite>+++

image::{{<siteurl>}}/uploads/2016/05/2016-05-22_09-52-16.png[2016 05 22 09 52 16,400]

I wanted to add a 'nationality' property to the players table in the SQL schema and to the nodes with the 'Player' label in the graph.

This refactoring is quite easy in both models. In the relational database we first run a query to add the 'nationality' field to the table:

[source,sql]
----

ALTER TABLE players
ADD COLUMN nationality varying(30);
----

And then we need to generate UPDATE statements from our data dump to update all the existing records:

[source,sql]
----

UPDATE players
SET nationality = 'Brazil'
WHERE players.id = '/aldair/profil/spieler/4151';

...
----

In the graph we can do this in a single step by processing our data dump using the LOAD CSV command and then setting a property on each player:

[source,cypher]
----

USING PERIODIC COMMIT
LOAD CSV WITH HEADERS FROM "file:///transfers.csv" AS row
MATCH (player:Player {id: row.playerUri})
SET player.nationality = row.playerNationality
----

If we wanted to make the nationality field non nullable we could go back and run the following queries:

[source,sql]
----

ALTER TABLE players
ALTER COLUMN nationality SET NOT NULL
----

[source,cypher]
----

CREATE CONSTRAINT ON (player:Player)
ASSERT exists(player.nationality)
----

And we're done!
Neo4j: Find the intermediate point between two lat/longs
Date: 2016-11-01 22:10:57

Yesterday I wrote a blog post showing how to find the midpoint between two lat/longs using Cypher which worked well as a first attempt at filling in missing locations, but I realised I could do better.

As I mentioned in the last post, when I find a stop that's missing lat/long coordinates I can usually find two nearby stops that allow me to triangulate this stop's location.

I also have train routes which indicate the number of seconds it takes to go from one stop to another, which allows me to indicate whether the location-less stop is closer to one stop than the other.

For example, consider stops a, b, and c where b doesn't have a location. If we have these distances between the stops:

[source,text]
----

(a)-[:NEXT {time: 60}]->(b)-[:NEXT {time: 240}]->(c)
----

it tells us that point 'b' is actually 0.2 of the distance from 'a' to 'c' rather than being the midpoint.

There's http://www.movable-type.co.uk/scripts/latlong.html#intermediate-point[a formula] we can use to work out that point:

[source,text]
----

a = sin((1−f)⋅δ) / sin δ
b = sin(f⋅δ) / sin δ
x = a ⋅ cos φ1 ⋅ cos λ1 + b ⋅ cos φ2 ⋅ cos λ2
y = a ⋅ cos φ1 ⋅ sin λ1 + b ⋅ cos φ2 ⋅ sin λ2
z = a ⋅ sin φ1 + b ⋅ sin φ2
φi = atan2(z, √x² + y²)
λi = atan2(y, x)

δ is the angular distance d/R between the two points.
φ = latitude
λ = longitude
----

Translated to Cypher (with mandatory Greek symbols) it reads like this to find the point 0.2 of the way from one point to another

~~~cypher with {latitude: 51.4931963543, longitude: -0.0475185810} AS p1, {latitude: 51.47908, longitude: -0.05393950 } AS p2 WITH p1, p2, distance(point(p1), point(p2)) / 6371000 AS δ, 0.2 AS f WITH p1, p2, δ, sin((1-f) * δ) / sin(δ) AS a, sin(f * δ) / sin(δ) AS b WITH radians(p1.latitude) AS φ1, radians(p1.longitude) AS λ1, radians(p2.latitude) AS φ2, radians(p2.longitude) AS λ2, a, b WITH a * cos(φ1) * cos(λ1) + b * cos(φ2) * cos(λ2) AS x, a * cos(φ1) * sin(λ1) + b * cos(φ2) * sin(λ2) AS y, a * sin(φ1) + b * sin(φ2) AS z RETURN degrees(atan2(z, sqrt(x{caret}2 + y{caret}2))) AS φi, degrees(atan2(y,x)) AS λi ~~~ ~~~text ╒═════════════════╤════════════════════╕ │φi │λi │ ╞═════════════════╪════════════════════╡ │51.49037311149128│-0.04880308288561931│ └─────────────────┴────────────────────┘ ~~~

A quick sanity check plugging in 0.5 instead of 0.2 finds the midpoint which I was able to sanity check against yesterday's post: ~~~text ╒═════════════════╤═════════════════════╕ │φi │λi │ ╞═════════════════╪═════════════════════╡ │51.48613822097523│-0.050729537454086385│ └─────────────────┴─────────────────────┘ ~~~

That's all for now!
Kubernetes: Writing hostname to a file
Date: 2016-11-22 19:56:31

Over the weekend I spent a bit of time playing around with Kubernetes and to get the hang of the technology I set myself the task of writing the hostname of the machine to a file.

I'm using the excellent https://github.com/kubernetes/minikube[minikube] tool to create a local Kubernetes cluster for my experiments so the first step is to spin that up:

[source,bash]
----

$ minikube start
Starting local Kubernetes cluster...
Kubectl is now configured to use the cluster.
----

The first thing I needed to work out how to get the hostname. I figured there was probably an environment variable that I could access. We can call the +++<cite>+++env+++</cite>+++ command to see a list of all the environment variables in a container so I created a pod template that would show me that information:

+++<cite>+++hostname_super_simple.yaml+++</cite>+++

[source,text]
----

apiVersion: v1
kind: Pod
metadata:
  name: mark-super-simple-test-pod
spec:
  containers:
    - name: test-container
      image: gcr.io/google_containers/busybox:1.24
      command: [ "/bin/sh", "-c", "env" ]
  dnsPolicy: Default
  restartPolicy: Never
----

I then created a pod from that template and checked the logs of that pod:

[source,bash]
----

$ kubectl create -f hostname_super_simple.yaml
pod "mark-super-simple-test-pod" created
----

[source,bash]
----

$ kubectl logs  mark-super-simple-test-pod
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT=tcp://10.0.0.1:443
HOSTNAME=mark-super-simple-test-pod
SHLVL=1
HOME=/root
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
PWD=/
KUBERNETES_SERVICE_HOST=10.0.0.1
----

The information we need is in +++<cite>+++$HOSTNAME+++</cite>+++ so the next thing we need to do is created a pod template which puts that into a file.

+++<cite>+++hostname_simple.yaml+++</cite>+++

[source,text]
----

apiVersion: v1
kind: Pod
metadata:
  name: mark-test-pod
spec:
  containers:
    - name: test-container
      image: gcr.io/google_containers/busybox:1.24
      command: [ "/bin/sh", "-c", "echo $HOSTNAME > /tmp/bar; cat /tmp/bar" ]
  dnsPolicy: Default
  restartPolicy: Never
----

We can create a pod using this template by running the following command:

[source,bash]
----

$ kubectl create -f hostname_simple.yaml
pod "mark-test-pod" created
----

Now let's check the logs of the instance to see whether our script worked:

[source,bash]
----

$ kubectl logs mark-test-pod
mark-test-pod
----

Indeed it did, good times!
Neo4j: Procedure call inside a query does not support passing arguments implicitly (pass explicitly after procedure name instead)
Date: 2016-10-02 10:13:26

A couple of days I was trying to write a Cypher query to filter the labels in my database.

I started with the following procedure call to get the list of all the labels:

[source,cypher]
----

CALL db.labels
----

[source,text]
----

╒══════════╕
│label     │
╞══════════╡
│Airport   │
├──────────┤
│Flight    │
├──────────┤
│Airline   │
├──────────┤
│Movie     │
├──────────┤
│AirportDay│
├──────────┤
│Person    │
├──────────┤
│Engineer  │
└──────────┘
----

I was only interested in labels that contained the letter 'a' so I tweaked the query to filter the output of the procedure:

[source,cypher]
----

CALL db.labels
YIELD label
WITH label WHERE tolower(label) contains "a"
RETURN label
----

Unfortunately that didn't work as I expected:

[source,text]
----

Procedure call inside a query does not support passing arguments implicitly (pass explicitly after procedure name instead) (line 1, column 9 (offset: 8))
"CALL db.labels"
         ^
----

The mistake I made was calling the procedure implicitly without using parentheses. If you want to do any post processing on the output of a procedure you need to call it explicitly otherwise Cypher gets very confused.

If we add back the parentheses it's much happier:

[source,cypher]
----

CALL db.labels()
YIELD label
WITH label WHERE tolower(label) contains "a"
RETURN label
----

[source,text]
----

╒══════════╕
│label     │
╞══════════╡
│Airport   │
├──────────┤
│Airline   │
├──────────┤
│AirportDay│
└──────────┘
----

It stumped me for a while until I figured out what the error message meant! I think I'll just use explicit parentheses all the time from now on to save me running into this one again.
Neo4j: Detecting rogue spaces in CSV headers with LOAD CSV
Date: 2016-10-19 05:16:07

Last week I was helping someone load the data from a CSV file into Neo4j and we were having trouble filtering out rows which contained a null value in one of the columns.

This is what the data looked like:

[source,cypher]
----

load csv with headers from "file:///foo.csv" as row
RETURN row
----

[source,text]
----

╒══════════════════════════════════╕
│row                               │
╞══════════════════════════════════╡
│{key1: a,  key2: (null),  key3: c}│
├──────────────────────────────────┤
│{key1: d,  key2: e,  key3: f}     │
└──────────────────────────────────┘
----

We'd like to filter out any rows which have 'key2' as null, so let's tweak our query to do that:

[source,cypher]
----

load csv with headers from "file:///foo.csv" as row
WITH row WHERE NOT row.key2 is null
RETURN row
----

[source,text]
----

(no rows)
----

Hmmm that's odd, it's got rid of both rows. We'd expect to see the 2nd row since that doesn't have a null value.

At this point we might suspect that what we're seeing on the screen isn't actually what the data looks like. Let's write the following query to check our header values:

[source,cypher]
----

load csv with headers from "file:///foo.csv" as row
WITH row LIMIT 1
UNWIND keys(row) AS key
RETURN key, SIZE(key)
----

[source,text]
----

╒═════╤═════════╕
│key  │SIZE(key)│
╞═════╪═════════╡
│key1 │4        │
├─────┼─────────┤
│ key2│5        │
├─────┼─────────┤
│ key3│5        │
└─────┴─────────┘
----

The second column tells us that there are some extra characters in the columns for 'key2' and 'key3' or rather ' key2' and ' key3'. In this case they are spaces, but it could easily be another character:

[source,cypher]
----

load csv with headers from "file:///foo.csv" as row
WITH row LIMIT 1
UNWIND keys(row) AS key
RETURN key, replace(key, " ", "_SPACE_") AS spaces
----

[source,text]
----

╒═════╤═══════════╕
│key  │spaces     │
╞═════╪═══════════╡
│key1 │key1       │
├─────┼───────────┤
│ key2│_SPACE_key2│
├─────┼───────────┤
│ key3│_SPACE_key3│
└─────┴───────────┘
----

If we clean up our CSV file and try again everything works as expected:

[source,cypher]
----

load csv with headers from "file:///foo.csv" as row
WITH row LIMIT 1
UNWIND keys(row) AS key
RETURN key, SIZE(key)
----

[source,text]
----

╒════╤═════════╕
│key │SIZE(key)│
╞════╪═════════╡
│key1│4        │
├────┼─────────┤
│key2│4        │
├────┼─────────┤
│key3│4        │
└────┴─────────┘
----

[source,cypher]
----

load csv with headers from "file:///foo.csv" as row
WITH row WHERE NOT row.key2 is null
RETURN row
----

[source,text]
----

╒═══════════════════════════╕
│row                        │
╞═══════════════════════════╡
│{key1: d, key2: e, key3: f}│
└───────────────────────────┘
----