markhneedham/blog huggingface dataset:
R: Refactoring to dplyr
Date: 2014-11-09 00:11:48

I've been looking back over some of the early code I wrote using R before I knew about the http://cran.r-project.org/web/packages/dplyr/vignettes/introduction.html[dplyr] library and thought it'd be an interesting exercise to refactor some of the snippets.

We'll use the following data frame for each of the examples:

[source,r]
----

library(dplyr)

data = data.frame(
  letter = sample(LETTERS, 50000, replace = TRUE),
  number = sample (1:10, 50000, replace = TRUE)
  )
----

== Take \{n} rows

[source,r]
----

> data[1:5,]
  letter number
1      R      7
2      Q      3
3      B      8
4      R      3
5      U      2
----

becomes:

[source,r]
----

> data %>% head(5)
  letter number
1      R      7
2      Q      3
3      B      8
4      R      3
5      U      2
----

== Order by numeric value descending

[source,r]
----

> data[order(-(data$number)),][1:5,]
   letter number
14      H     10
17      G     10
63      L     10
66      W     10
73      R     10
----

becomes:

[source,r]
----

> data %>% arrange(desc(number)) %>% head(5)
  letter number
1      H     10
2      G     10
3      L     10
4      W     10
5      R     10
----

== Count number of items

[source,r]
----

> length(data[,1])
[1] 50000
----

becomes:

[source,r]
----

> data %>% count()
Source: local data frame [1 x 1]

      n
1 50000
----

== Filter by column value

[source,r]
----

> length(subset(data, number == 1)[, 1])
[1] 4928
----

becomes:

[source,r]
----

> data %>% filter(number == 1) %>% count()
Source: local data frame [1 x 1]

     n
1 4928
----

== Group by variable and count

[source,r]
----

> aggregate(data, by= list(data$number), function(x) length(x))
   Group.1 letter number
1        1   4928   4928
2        2   5045   5045
3        3   5064   5064
4        4   4823   4823
5        5   5032   5032
6        6   5163   5163
7        7   4945   4945
8        8   5077   5077
9        9   5025   5025
10      10   4898   4898
----

becomes:

[source,r]
----

> data %>% count(number)
Source: local data frame [10 x 2]

   number    n
1       1 4928
2       2 5045
3       3 5064
4       4 4823
5       5 5032
6       6 5163
7       7 4945
8       8 5077
9       9 5025
10     10 4898
----

== Select a range of rows

[source,r]
----

> data[4:5,]
  letter number
4      R      3
5      U      2
----

becomes:

[source,r]
----

> data %>% slice(4:5)
  letter number
1      R      3
2      U      2
----

There's certainly more code in some of the dplyr examples but I find it easier to remember how the dplyr code works when I come back to it and hence tend to favour that approach.
R: dplyr - "Variables not shown"
Date: 2014-11-23 01:02:06

I recently ran into a problem where the result of applying some operations to a data frame wasn't being output the way I wanted.

I started with this data frame:

[source,r]
----

words = function(numberOfWords, lengthOfWord) {
  w = c(1:numberOfWords)
  for(i in 1:numberOfWords) {
    w[i] = paste(sample(letters, lengthOfWord, replace=TRUE), collapse = "")
  }
  w
}

numberOfRows = 100
df = data.frame(a = sample (1:numberOfRows, 10, replace = TRUE),
                b = sample (1:numberOfRows, 10, replace = TRUE),
                name = words(numberOfRows, 10))
----

I wanted to group the data frame by +++<cite>+++a+++</cite>+++ and +++<cite>+++b+++</cite>+++ and output a comma separated list of the associated names. I started with this:

[source,r]
----

> df %>%
    group_by(a,b) %>%
    summarise(n = n(), words = paste(name, collapse = ",")) %>%
    arrange(desc(n)) %>%
    head(5)

Source: local data frame [5 x 4]
Groups: a

   a  b  n
1 19 90 10
2 24 36 10
3 29 20 10
4 29 80 10
5 62 54 10
Variables not shown: words (chr)
----

Unfortunately the +++<cite>+++words+++</cite>+++ column has been excluded and I came across http://stackoverflow.com/questions/22471256/overwriting-variables-not-shown-in-dplyr[this Stack Overflow post] which suggested that the +++<cite>+++print.tbl_df+++</cite>+++ function was the one responsible for filtering columns.

Browsing the docs I found a couple of ways to overwrite this behaviour:

[source,r]
----

> df %>%
    group_by(a,b) %>%
    summarise(n = n(), words = paste(name, collapse = ",")) %>%
    arrange(desc(n)) %>%
    head(5) %>%
    print(width = Inf)
----

or

[source,r]
----

> options(dplyr.width = Inf)
> df %>%
    group_by(a,b) %>%
    summarise(n = n(), words = paste(name, collapse = ",")) %>%
    arrange(desc(n)) %>%
    head(5)
----

And now we see this output instead:

[source,r]
----

Source: local data frame [5 x 4]
Groups: a

   a  b  n                                                                                                         words
1 19 90 10 dfhtcgymxt,zpemxbpnri,rfmkksuavp,jxaarxzdzd,peydpxjizc,trdzchaxiy,arthnxbaeg,kjbpdvvghm,kpvsddlsua,xmysfcynxw
2 24 36 10 wtokzdfecx,eprsvpsdcp,kzgxtwnqli,jbyuicevrn,klriuenjzu,qzgtmkljoy,bonbhmqfaz,uauoybprrl,rzummfbkbx,icyeorwzxl
3 29 20 10 ebubytlosp,vtligdgvqw,ejlqonhuit,jwidjvtark,kmdzcalblg,qzrlewxcsr,eckfgjnkys,vfdaeqbfqi,rumblliqmn,fvezcdfiaz
4 29 80 10 wputpwgayx,lpawiyhzuh,ufykwguynu,nyqnwjallh,abaxicpixl,uirudflazn,wyynsikwcl,usescualww,bkvsowfaab,gfhyifzepx
5 62 54 10 beuegfzssp,gfmegjtrys,wkubhvnkkk,rkhgprxttb,cwsrzulnpo,hzkvjbiywc,gbmiupnlbw,gffovxwtok,uxadfrjvdn,aojjfhxygs
----

Much better!
Python: Converting a date string to timestamp
Date: 2014-10-20 15:53:51

I've been playing around with Python over the last few days while cleaning up a data set and one thing I wanted to do was translate date strings into a timestamp.

I started with a date in this format:

[source,python]
----

date_text = "13SEP2014"
----

So the first step is to translate that into a Python date - the https://docs.python.org/2/library/datetime.html#strftime-strptime-behavior[strftime section of the documentation] is useful for figuring out which format code is needed:

[source,python]
----

import datetime

date_text = "13SEP2014"
date = datetime.datetime.strptime(date_text, "%d%b%Y")

print(date)
----

[source,bash]
----

$ python dates.py
2014-09-13 00:00:00
----

The next step was to translate that to a UNIX timestamp. I thought there might be a method or property on the Date object that I could access but I couldn't find one and so ended up using https://docs.python.org/2/library/calendar.html#calendar.timegm[calendar] to do the transformation:

[source,python]
----

import datetime
import calendar

date_text = "13SEP2014"
date = datetime.datetime.strptime(date_text, "%d%b%Y")

print(date)
print(calendar.timegm(date.utctimetuple()))
----

[source,text]
----

$ python dates.py
2014-09-13 00:00:00
1410566400
----

It's not too tricky so hopefully I shall remember next time.
Data Modelling: The Thin Model
Date: 2014-10-27 06:55:34

About a third of the way through http://www.amazon.co.uk/Mastering-Data-Modeling-Driven-Approach/dp/020170045X/ref=sr_1_1?ie=UTF8&qid=1414364532&sr=8-1&keywords=mastering+data+modeling[Mastering Data Modeling] the authors describe common data modelling mistakes and one in particular resonated with me - '*Thin LDS, Lost Users*'.

LDS stands for 'Logical Data Structure' which is a diagram depicting what kinds of data some person or group wants to remember. In other words, a tool to help derive the http://www.markhneedham.com/blog/2014/10/06/conceptual-model-vs-graph-model/[conceptual model] for our domain.

They describe the problem that a thin model can cause as follows:

____
[\...] within 30 minutes [of the modelling session] the users were lost\...we determined that the model was too thin. That is, many entities had just identifying descriptors. While this is syntactically okay, when we revisited those entities asking, _What else is memorable here?_ the users had lots to say. When there was flesh on the bones, the uncertainty abated and the session took a positive course.
____

I found myself making the same mistake a couple of weeks ago during a graph modelling session. I tend to spend the majority of the time focused on the relationships between the bits of data and treat the meta data or attributes almost as an after thought.

image::{{<siteurl>}}/uploads/2014/10/2014-10-27_06-41-19.png[2014 10 27 06 41 19,300]

The nice thing about the graph model is that it encourages an iterative approach so I was quickly able to bring the model to life and the domain experts back onside.

We can see a simple example of adding flesh to a model with a subset of the movies graph.

We might start out with the model on the right hand side which just describes the structure of the graph but doesn't give us very much information about the entities.

I tend to sketch out the structure of all the data before adding any attributes but I think some people find it easier to follow if you add at least some flesh before moving on to the next part of the model.

In our next iteration of the movie graph we can add attributes to the actor and movie:

image::{{<siteurl>}}/uploads/2014/10/2014-10-27_06-57-32.png[2014 10 27 06 57 32,600]

We can then go on to evolve the model further but the lesson for me is *value the attributes more*, it's not all about the structure.
R: Converting a named vector to a data frame
Date: 2014-10-31 23:47:26

I've been playing around with http://igraph.org/r/[igraph's] http://igraph.org/r/doc/page.rank.html[page rank] function to see who the most central nodes in the London NoSQL scene are and I wanted to put the result in a data frame to make the data easier to work with.

I started off with a data frame containing pairs of people and the number of events that they'd both RSVP'd 'yes' to:

[source,r]
----

> library(dplyr)
> data %>% arrange(desc(times)) %>% head(10)
       p.name     other.name times
1  Amit Nandi Anish Mohammed    51
2  Amit Nandi Enzo Martoglio    49
3       louis          zheng    46
4       louis     Raja Kolli    45
5  Raja Kolli Enzo Martoglio    43
6  Amit Nandi     Raja Kolli    42
7       zheng Anish Mohammed    42
8  Raja Kolli          Rohit    41
9  Amit Nandi          zheng    40
10      louis          Rohit    40
----

I actually had ~ 900,000 such rows in the data frame:

[source,r]
----

> length(data[,1])
[1] 985664
----

I ran page rank over the data set like so:

[source,r]
----

g = graph.data.frame(data, directed = F)
pr = page.rank(g)$vector
----

If we evaluate +++<cite>+++pr+++</cite>+++ we can see the person's name and their page rank:

[source,r]
----

> head(pr)
Ioanna Eirini          Mjay       Baktash      madhuban    Karl Prior   Keith Bolam
    0.0002190     0.0001206     0.0001524     0.0008819     0.0001240     0.0005702
----

I initially tried to convert this to a data frame with the following code\...

[source,r]
----

> head(data.frame(pr))
                     pr
Ioanna Eirini 0.0002190
Mjay          0.0001206
Baktash       0.0001524
madhuban      0.0008819
Karl Prior    0.0001240
Keith Bolam   0.0005702
----

\...which unfortunately didn't create a column for the person's name.

[source,R]
----

> colnames(data.frame(pr))
[1] "pr"
----

http://nicolewhite.github.io/[Nicole] pointed out that I actually had a http://www.r-tutor.com/r-introduction/vector/named-vector-members[named vector] and would need to explicitly extract the names from that vector into the data frame. I ended up with this:

[source,r]
----

> prDf = data.frame(name = names(pr), rank = pr)
> head(prDf)
                       name      rank
Ioanna Eirini Ioanna Eirini 0.0002190
Mjay                   Mjay 0.0001206
Baktash             Baktash 0.0001524
madhuban           madhuban 0.0008819
Karl Prior       Karl Prior 0.0001240
Keith Bolam     Keith Bolam 0.0005702
----

We can now sort the data frame to find the most central people on the NoSQL London scene based on meetup attendance:

[source,r]
----

> data.frame(prDf) %>%
+   arrange(desc(pr)) %>%
+   head(10)
             name     rank
1           louis 0.001708
2       Kannappan 0.001657
3           zheng 0.001514
4    Peter Morgan 0.001492
5      Ricki Long 0.001437
6      Raja Kolli 0.001416
7      Amit Nandi 0.001411
8  Enzo Martoglio 0.001396
9           Chris 0.001327
10          Rohit 0.001305
----
hdiutil: could not access / create failed - Operation canceled
Date: 2014-10-31 09:45:32

Earlier in the year I wrote a http://www.markhneedham.com/blog/2014/04/07/install4j-and-applescript-creating-a-mac-os-x-application-bundle-for-a-java-application/[blog post showing how to build a Mac OS X DMG file for a Java application] and I recently revisited this script to update it to a new version and ran into a frustrating error message.

I tried to run the following command to create a new DMG file from a source folder\...

[source,text]
----

$ hdiutil create -volname "DemoBench" -size 100m -srcfolder dmg/ -ov -format UDZO pack.temp.dmg
----

\...but was met with the following error message: ~~~text \...could not access /Volumes/DemoBench/DemoBench.app/Contents/Resources/app/database-agent.jar - Operation canceled hdiutil: create failed - Operation canceled ~~~

I was initially a bit stumped and thought maybe the flags to hdiutil had changed but a quick look at the man page suggested that wasn't the issue.

I decided to go back to my pre command line approach for creating a DMG - DiskUtility - and see if I could create it that way. This helped reveal the actual problem:

image::{{<siteurl>}}/uploads/2014/10/2014-10-31_09-42-20.png[2014 10 31 09 42 20,600]

I increased the volume size to 150 MB\... ~~~text $ hdiutil create -volname "DemoBench" -size 150m -srcfolder dmg/ -ov -format UDZO pack.temp.dmg ~~~

and all was well: ~~~text \...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\.... \...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\...\..... created: /Users/markneedham/projects/neo-technology/quality-tasks/park-bench/database-agent-desktop/target/pack.temp.dmg ~~~

And this post will serve as documentation to stop it catching me out next time!
4 types of user
Date: 2014-07-29 19:07:11

I've been working with Neo4j full time for slightly more than a year now and from interacting with the community I've noticed that while using different features of the product people fall into 4 categories.

These are as follows:

image::{{<siteurl>}}/uploads/2014/07/4types.jpg[4types,400]

On one axis we have 'loudness' i.e. how vocal somebody is either on twitter, StackOverflow or by email and on the other we have 'success' which is how well a product feature is working for them.

The people in the top half of the diagram will get the most attention because they're the most visible.

Of those people we'll tend to spend more time on the people who are unhappy and vocal to try and help them solve the problems their having.

When working with the people in the top left it's difficult to understand how representative they are for the whole user base.

It could be the case that they aren't representative at all and actually there is a quiet majority who the product is working for and are just getting on with it with no fuss.

However, it could equally be the case that they are *absolutely representative* and there are a lot of users quietly suffering / giving up using the product.

I haven't come up with a good way to come across the less vocal users but in my experience they'll often be passive users of the user group or Stack Overflow i.e. they'll read existing issues but not post anything themselves.

Given this uncertainty I think it makes sense to assume that the silent majority suffer the same problems as the more vocal minority.

Another interesting thing I've noticed about this quadrant is that the people in the top right are often the best people in the community to help those who are struggling.

It'd be interesting to know whether anyone has noticed a similar thing with the products they worked on, and if so what approach do you take to unveiling the silent majority?
R: Apply a custom function across multiple lists
Date: 2014-07-16 05:04:46

In my continued playing around with http://www.r-project.org/[R] I wanted to map a custom function over two lists comparing each item with its corresponding items.

If we just want to use a built in function such as subtraction between two lists it's quite easy to do:

[source,r]
----

> c(10,9,8,7,6,5,4,3,2,1) - c(5,4,3,4,3,2,2,1,2,1)
 [1] 5 5 5 3 3 3 2 2 0 0
----

I wanted to do a slight variation on that where instead of returning the difference I wanted to return a text value representing the difference e.g. '5 or more', '3 to 5' etc.

I spent a long time trying to figure out how to do that before finding http://nsaunders.wordpress.com/2010/08/20/a-brief-introduction-to-apply-in-r/[an excellent blog post which describes all the different 'apply' functions available in R].

As far as I understand 'apply' is the equivalent of 'map' in Clojure or other functional languages.

In this case we want the +++<cite>+++http://stat.ethz.ch/R-manual/R-patched/library/base/html/mapply.html[mapply]+++</cite>+++ variant which we can use like so:

[source,r]
----

> mapply(function(x, y) {
    if((x-y) >= 5) {
        "5 or more"
    } else if((x-y) >= 3) {
        "3 to 5"
    } else {
        "less than 5"
    }
  }, c(10,9,8,7,6,5,4,3,2,1),c(5,4,3,4,3,2,2,1,2,1))
 [1] "5 or more"   "5 or more"   "5 or more"   "3 to 5"      "3 to 5"      "3 to 5"      "less than 5"
 [8] "less than 5" "less than 5" "less than 5"
----

We could then pull that out into a function if we wanted:

[source,r]
----

summarisedDifference <- function(one, two) {
  mapply(function(x, y) {
    if((x-y) >= 5) {
      "5 or more"
    } else if((x-y) >= 3) {
      "3 to 5"
    } else {
      "less than 5"
    }
  }, one, two)
}
----

which we could call like so:

[source,r]
----

> summarisedDifference(c(10,9,8,7,6,5,4,3,2,1),c(5,4,3,4,3,2,2,1,2,1))
 [1] "5 or more"   "5 or more"   "5 or more"   "3 to 5"      "3 to 5"      "3 to 5"      "less than 5"
 [8] "less than 5" "less than 5" "less than 5"
----

I also wanted to be able to compare a list of items to a single item which was much easier than I expected:

[source,r]
----

> summarisedDifference(c(10,9,8,7,6,5,4,3,2,1), 1)
 [1] "5 or more"   "5 or more"   "5 or more"   "5 or more"   "5 or more"   "3 to 5"      "3 to 5"
 [8] "less than 5" "less than 5" "less than 5"
----

If we wanted to get a summary of the differences between the lists we could plug them into ddply like so:

[source,r]
----

> library(plyr)
> df = data.frame(x=c(10,9,8,7,6,5,4,3,2,1), y=c(5,4,3,4,3,2,2,1,2,1))
> ddply(df, .(difference=summarisedDifference(x,y)), summarise, count=length(x))
   difference count
1      3 to 5     3
2   5 or more     3
3 less than 5     4
----
R/plyr: ddply - Error in vector(type, length) : vector: cannot make a vector of mode 'closure'.
Date: 2014-07-07 06:07:29

In my continued playing around with plyr's http://www.inside-r.org/packages/cran/plyr/docs/ddply[ddply] function I was trying to group a data frame by one of its columns and return a count of the number of rows with specific values and ran into a strange (to me) error message.

I had a data frame:

[source,r]
----

n = c(2, 3, 5)
s = c("aa", "bb", "cc")
b = c(TRUE, FALSE, TRUE)
df = data.frame(n, s, b)
----

And wanted to group and count on column 'b' so I'd get back a count of 2 for TRUE and 1 for FALSE. I wrote this code:

[source,r]
----

ddply(df, "b", function(x) {
  countr <- length(x$n)
  data.frame(count = count)
})
----

which when evaluated gave the following error:

[source,r]
----

Error in vector(type, length) :
  vector: cannot make a vector of mode 'closure'.
----

It took me quite a while to realise that I'd just made a typo in assigned the count to a variable called 'countr' instead of 'count'.

As a result of that typo I think the R compiler was trying to find a variable called 'count' somwhere else in the lexical scope but was unable to. If I'd defined the variable 'count' outside the call to ddply function then my typo wouldn't have resulted in an error but rather an unexpected resulte.g.

[source,r]
----

> count = 10
----

[source,r]
----

> ddply(df, "b", function(x) {
+   countr <- length(x$n)
+   data.frame(count = count)
+ })
      b count
1 FALSE     4
2  TRUE     4
----

Once I spotted the typo and fixed it things worked as expected:

[source,r]
----

> ddply(df, "b", function(x) {
+   count <- length(x$n)
+   data.frame(count = count)
+ })
----

[source,r]
----

      b count
1 FALSE     1
2  TRUE     2
----
Java: Determining the status of data import using kill signals
Date: 2014-07-23 22:20:25

A few weeks ago I was working on the initial import of ~ 60 million bits of data into Neo4j and we kept running into a problem where the import process just seemed to freeze and nothing else was imported.

It was very difficult to tell what was happening inside the process - taking a thread dump merely informed us that it was attempting to process one line of a CSV line and was somehow unable to do so.

One way to help debug this would have been to print out every single line of the CSV as we processed it and then watch where it got stuck but this seemed a bit over kill. Ideally we wanted to only print out the line we were processing on demand.

As luck would have it we can do exactly this by sending a kill signal to our import process and have it print out where it had got up to. We had to make sure we picked a signal which wasn't already being handled by the JVM and decided to go with 'SIGTRAP' i.e. kill -5 [pid]

We came across http://www.javawebdevelop.com/300909/[a neat blog post that explained how to wire everything up] and then created our own version:

[source,java]
----

class Kill3Handler implements SignalHandler
{
    private AtomicInteger linesProcessed;
    private AtomicReference<Map<String, Object>> lastRowProcessed;

    public Kill3Handler( AtomicInteger linesProcessed, AtomicReference<Map<String, Object>> lastRowProcessed )
    {
        this.linesProcessed = linesProcessed;
        this.lastRowProcessed = lastRowProcessed;
    }

    @Override
    public void handle( Signal signal )
    {
        System.out.println("Last Line Processed: " + linesProcessed.get() + " " + lastRowProcessed.get());
    }
}
----

We then wired that up like so:

[source,java]
----

AtomicInteger linesProcessed = new AtomicInteger( 0 );
AtomicReference<Map<String, Object>> lastRowProcessed = new AtomicReference<>(  );
Kill3Handler kill3Handler = new Kill3Handler( linesProcessed, lastRowProcessed );
Signal.handle(new Signal("TRAP"), kill3Handler);

// as we iterate each line we update those variables

linesProcessed.incrementAndGet();
lastRowProcessed.getAndSet( properties ); // properties = a representation of the row we're processing
----

This worked really well for us and we were able to work out that we had a slight problem with some of the data in our CSV file which was causing it to be processed incorrectly.

We hadn't been able to see this by visual inspection since the CSV files were a few GB in size. We'd therefore only skimmed a few lines as a sanity check.

I didn't even know you could do this but it's a neat trick to keep in mind - I'm sure it shall come in useful again.