markhneedham/blog huggingface dataset:
R: Regex - capturing multiple matches of the same group
Date: 2015-06-19 21:38:47

I've been playing around with some web logs using R and I wanted to extract everything that existed in double quotes within a logged entry.

This is an example of a log entry that I want to parse:

[source,r]
----

log = '2015-06-18-22:277:548311224723746831\t2015-06-18T22:00:11\t2015-06-18T22:00:05Z\t93317114\tip-127-0-0-1\t127.0.0.5\tUser\tNotice\tneo4j.com.access.log\t127.0.0.3 - - [18/Jun/2015:22:00:11 +0000] "GET /docs/stable/query-updating.html HTTP/1.1" 304 0 "http://neo4j.com/docs/stable/cypher-introduction.html" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36"'
----

And I want to extract these 3 things:

* /docs/stable/query-updating.html
* http://neo4j.com/docs/stable/cypher-introduction.html
* Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36

i.e. the URI, the referrer and browser details.

I'll be using the +++<cite>+++stringr+++</cite>+++ library which seems to work quite well for this type of work.

To extract these values we need to find all the occurrences of double quotes and get the text inside those quotes. We might start by using the +++<cite>+++str_match+++</cite>+++ function:

[source,r]
----

> library(stringr)
> str_match(log, "\"[^\"]*\"")
     [,1]
[1,] "\"GET /docs/stable/query-updating.html HTTP/1.1\""
----

Unfortunately that only picked up the first occurrence of the pattern so we've got the URI but not the referrer or browser details. I tried +++<cite>+++str_extract+++</cite>+++ with similar results before I found +++<cite>+++str_extract_all+++</cite>+++ which does the job:

[source,r]
----

> str_extract_all(log, "\"[^\"]*\"")
[[1]]
[1] "\"GET /docs/stable/query-updating.html HTTP/1.1\""
[2] "\"http://neo4j.com/docs/stable/cypher-introduction.html\""
[3] "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36\""
----

We still need to do a bit of cleanup to get rid of the 'GET' and 'HTTP/1.1' in the URI and the quotes in all of them:

[source,r]
----

parts = str_extract_all(log, "\"[^\"]*\"")[[1]]
uri = str_match(parts[1], "GET (.*) HTTP")[2]
referer = str_match(parts[2], "\"(.*)\"")[2]
browser = str_match(parts[3], "\"(.*)\"")[2]

> uri
[1] "/docs/stable/query-updating.html"

> referer
[1] "https://www.google.com/"

> browser
[1] "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.124 Safari/537.36"
----

We could then go on to split out the browser string into its sub components but that'll do for now!
R: write.csv - unimplemented type 'list' in 'EncodeElement'
Date: 2015-06-30 22:26:39

Everyone now and then I want to serialise an R data frame to a CSV file so I can easily load it up again if my R environment crashes without having to recalculate everything but recently ran into the following error:

[source,r]
----

> write.csv(foo, "/tmp/foo.csv", row.names = FALSE)
Error in .External2(C_writetable, x, file, nrow(x), p, rnames, sep, eol,  :
  unimplemented type 'list' in 'EncodeElement'
----

If we take a closer look at the data frame in question it looks ok:

[source,r]
----

> foo
  col1 col2
1    1    a
2    2    b
3    3    c
----

However, one of the columns contains a list in each cell and we need to find out which one it is. I've found the quickest way is to run the +++<cite>+++typeof+++</cite>+++ function over each column:

[source,r]
----

> typeof(foo$col1)
[1] "double"

> typeof(foo$col2)
[1] "list"
----

So 'col2' is the problem one which isn't surprising if you consider the way I created 'foo':

[source,r]
----

library(dplyr)
foo = data.frame(col1 = c(1,2,3)) %>% mutate(col2 = list("a", "b", "c"))
----

If we do have a list that we want to add to the data frame we need to convert it to a vector first so we don't run into this type of problem:

[source,R]
----

foo = data.frame(col1 = c(1,2,3)) %>% mutate(col2 = list("a", "b", "c") %>% unlist())
----

And now we can write to the CSV file:

[source,r]
----

write.csv(foo, "/tmp/foo.csv", row.names = FALSE)
----

[source,bash]
----

$ cat /tmp/foo.csv
"col1","col2"
1,"a"
2,"b"
3,"c"
----

And that's it!
R: Error in approxfun(x.values.1, y.values.1, method = "constant", f = 1, :  zero non-NA points
Date: 2015-12-27 12:24:05

I've been following http://www.r-bloggers.com/how-to-perform-a-logistic-regression-in-r/[Michy Alice's logistic regression tutorial] to create an attendance model for London dev meetups and ran into an interesting problem while doing so.

Our dataset has a class imbalance i.e. most people RSVP 'no' to events which can lead to misleading accuracy score where predicting 'no' every time would lead to supposed high accuracy.

[source,r]
----

Source: local data frame [2 x 2]

  attended     n
     (dbl) (int)
1        0  1541
2        1    53
----

I sampled the data using +++<cite>+++caret+++</cite>+++'s +++<cite>+++http://www.inside-r.org/packages/cran/caret/docs/upSample[upSample]+++</cite>+++ function to avoid this:

[source,r]
----

attended = as.factor((df %>% dplyr::select(attended))$attended)
upSampledDf = upSample(df %>% dplyr::select(-attended), attended)
upSampledDf$attended = as.numeric(as.character(upSampledDf$Class))
----

I then trained a logistic regression model but when I tried to plot the area under the curve I ran into trouble:

[source,r]
----

p <- predict(model, newdata=test, type="response")
pr <- prediction(p, test$attended)
prf <- performance(pr, measure = "tpr", x.measure = "fpr")

Error in approxfun(x.values.1, y.values.1, method = "constant", f = 1,  :
  zero non-NA points
----

I don't have any NA values in my data frame so this message was a bit confusing to start with. As usual http://stackoverflow.com/questions/23836955/error-in-approxfunx-values-1-y-values-1-method-constant-f-1-zero-no/33028711#33028711[Stack Overflow came to the rescue] with the suggestion that I was probably missing positive/negative values for the independent variable i.e. 'approved'.

A quick count on the test data frame using dplyr confirmed my mistake:

[source,r]
----

> test %>% count(attended)
Source: local data frame [1 x 2]

  attended     n
     (dbl) (int)
1        1   582
----

I'll have to http://stackoverflow.com/questions/9081498/the-correct-approach-of-randomly-re-ordering-a-matrix-in-r[randomly sort the data frame] and then reassign my training and test data frames to work around it.
Study until your mind wanders
Date: 2015-12-31 10:47:01

I've previously found it very difficult to read math heavy content which has made it challenging to read http://www.amazon.co.uk/Distributed-Computing-2e-Fundamentals-Simulations/dp/0471453242/ref=sr_1_1?ie=UTF8&qid=1451556471&sr=8-1&keywords=distributed+computing+fundamentals[Distributed Computing] which I bought last May.

After several false starts where I gave up after getting frustrated that I couldn't understand things the first time around and forgot everything if I left it a couple of days I decided to try again with a different approach.

I've been trying a technique http://www.markhneedham.com/blog/2015/03/17/one-month-of-mini-habits/[I learned] from http://www.amazon.co.uk/Mini-Habits-Smaller-Bigger-Results-ebook/dp/B00HGKNBDK[Mini Habits] where every day I have a (very small) goal of *reading one page of the book*. Having such a small goal means that I can read the material as slowly as I like (repeating previous days if necessary).

So far I've read 4 chapters (~100 pages) over the last month - some days I read 6 or 7 pages, other days I only manage one. The key is to keep the rhythm of reading something.

I tried doing the reading at different times of the day - on the bus on the way to work, in the evening before going to sleep - and found that for me the best time is immediately when I wake up. To minimise my mind wandering I don't read any emails, chat messages or social media accounts before I start.

Despite this, I've noticed that after a while *my mind starts to wander* while reading proofs and that's a signal for me to stop for the day. When I pick up again the next day I've often found that I understand what I was having difficulty with.

I've read that meditation prior to studying is an effective way to quiet the mind and would be a more 'on demand' way of achieving the concentration required to read this type of material. I've never done any meditation so if anyone has any tips on where to start that'd be helpful.
Haskell: Chaining functions to find the middle value in a collection
Date: 2012-03-20 23:36:03

I've been playing around with writing merge sort in Haskell and eventually ended up with the following function:

[source,haskell]
----

msort :: [Int] -> [Int]
msort unsorted = 	
  let n = floor (fromIntegral(length unsorted) / 2)
  in
    if n == 0 then unsorted
    else
      let (left, right) = splitAt n unsorted
      in merge (msort left) (msort right)
  where
    merge [] right = right
    merge left [] = left
    merge left@(x:xs) right@(y:ys) = if x < y then x : merge xs right else y : merge left ys
----

The 3rd line was annoying me as it has way too many brackets on it and I was fairly sure that it should be possible to just combine the functions http://www.markhneedham.com/blog/2009/01/12/f-partial-function-application-with-the-function-composition-operator/[like I learnt to do in F# a few years ago].

It's pretty easy to do that for the first two functions 'length' and 'fromIntegral' which we can do like this:

[source,haskell]
----

middle = fromIntegral . length
----

The third line now reads like this:

[source,haskell]
----

let n = floor ((middle unsorted) / 2)
----

It's a slight improvement but still not that great.

The problem with working out how to chain the division bit is that our value needs to be passed as the first argument to '/' so we can't do the following...

[source,haskell]
----

middle = ((/) 2) . fromIntegral . length
----

...since that divides 2 by the length of our collection rather than the other way around!
[source,haskell]
----

> middle [1,2,3,4,5,6]
0.3333333333333333
----

Instead we want to create an anonymous function around the '/' function and then apply floor:

[source,haskell]
----

middle :: [Int] -> Int
middle = floor . (\y -> y / 2) .  fromIntegral . length
----

And merge sort now looks like this:

[source,haskell]
----

msort :: [Int] -> [Int]
msort unsorted = 	
  let n = middle unsorted
  in
    if n == 0 then unsorted
    else
      let (left, right) = splitAt n unsorted
      in merge (msort left) (msort right)
  where
    merge [] right = right
    merge left [] = left
    merge left@(x:xs) right@(y:ys) = if x < y then x : merge xs right else y : merge left ys
----

Which I think is pretty neat!
Testing: Trying not to overdo it
Date: 2012-03-28 00:10:46

The design of the code which contains the main logic of the application that I'm currently working on looks a bit like the diagram on the right hand side:

image::{{<siteurl>}}/uploads/2012/03/orchestration-code.gif[Orchestration code,329]

We load a bunch of stuff from an Oracle database, construct some objects from the data and then invoke a sequence of methods on those objects in order to execute our domain logic.

Typically we might expect to see unit level test against all the classes described in this diagram but we've actually been trying out an approach where we don't test the orchestration code directly but rather only test it via the resource which makes use of it.

We originally started off writing some tests around that code but they ended up being really similar to our database and resource tests.

Having them around also made it difficult to change the way the orchestration worked since we'd end up breaking most of the tests when we tried to change anything.

One disadvantage of not testing this code is that we end up using the debugger more when trying to work out why resource tests aren't working since we now have more code being directly tested.

image::{{<siteurl>}}/uploads/2012/03/orchestration-tests2.gif[Orchestration tests2,427]

On the other hand we've been forced to drive logic into the domain objects as a result since we don't have any other place to test that functionality from.

Testing directly against the domain objects is much easier since everything's in memory and we can easily setup the data to be how we want it to be and inject it into the objects.

Another approach we could have taken would be to mock out the dependencies of the orchestration code but since this code is mostly coordinating other classes there are a lot of dependencies and the tests ended up being quite complicated and brittle.

Initially I was of the opinion that it wasn't a good idea to not test the orchestration code but looking back a month later I think it's working reasonably well and putting this constraint on ourselves has made the code easier to change while still being well tested.
Coding: Wait for the abstractions to emerge
Date: 2012-03-17 11:19:11

One of the things that I've learnt while developing code in an incremental way is that the way the code should be designed isn't going to be obvious straight away so we need to be patience and wait for it to emerge.

There's often a tendency to pull out classes or methods but more recently I've been trying to follow an approach where I leave the code in one class/method and play around with/study it until I see a good abstraction to make.

More often than not it takes a few times over multiple days before anything reveals itself.

I recently watched https://twitter.com/#!/venkat_s[Venkat Subramaniam's] Agile India presentation 'http://www.youtube.com/watch?v=iGsPeR-SYYo&feature=youtu.be[How to Approach Refactoring]' in which amongst other bits of advice he suggests that we '*make it work then make it evolve*' which sounds like it's covering the same type of ground.

Sometimes in the rush to abstract code to make it easier to test or to reduce the size of a class we end up creating an abstraction which doesn't really make sense in the business domain.

Having big methods of inline code goes against what we've learnt but putting all the code together makes it much easier to see the real abstractions in the code than if some premature abstractions have been made and we have to remember those as well.

It's still useful to be driven by the goal of pulling out small classes of behaviour/data that are easy to test but *spending a bit more time understanding the domain before doing so seems to work better*.

We recently had some unwieldy code which we couldn't quite work out how to abstract so we left the logic inline until we knew better even though it looked quite messy.

Eventually when talking to one of the subject matter experts they mentioned a term that they used which actually combined two things together which we'd originally thought were independent.

A simple rule of thumb is that if there is no name in the domain for what you're abstracting then it might not make sense to do so.

It doesn't always work but at least gets your thinking about whether you're improving the code with the abstraction you're about to make!
Functional Programming: One function at a time
Date: 2012-03-19 23:25:47

As I mentioned in an earlier post I got a bit stuck working out all the diagonals in the 20x20 grid of http://projecteuler.net/problem=11[Project Euler problem 11] and my http://www.markhneedham.com/blog/2012/03/13/functional-programming-shaping-the-data-to-fit-a-function/[colleague Uday ended up showing me how to do it].

I realised while watching him solve the problem that we'd been using quite different approaches to solving the problem and that his way worked way better than mine, at least in this context.

My approach was to try and drive out the solution from the top down which meant in this case:

* Get all of the diagonals left to right by iterating over a 2 dimensional array and accumulating sub arrays of the values 1 row and 1 column greater than the current position.

I was stuck for ages trying to work out how to do that whereas Uday made it much easier by just focusing on one of the functions that we'd need, getting that to work and then moving onto the next one.

After we'd iterated with that approach a few times we had a bunch of functions that we could compose together to solve the initial problem.

We started with a simple function to find the value at position (x,y) in the array:

[source,haskell]
----

findValue :: Int -> Int -> [[Int]] -> Int
findValue x y grid = (grid !! x) !! y
----

We then worked on writing a function in the REPL which would find all the diagonals going down from the current position

[source,haskell]
----

[findValue (0+z) (0+z) grid | z <- [0..19]]
----

We eventually realised that running that from anywhere but the top left hand corner would throw an exception since we'd gone out of bounds of the array:

[source,text]
----

> [findValue (1+z) (1+z) grid | z <- [0..19]]
[49,31,23,51,3,67,20,97,45,3,24,44,52,26,32,40,4,5,48,*** Exception: Prelude.(!!): index too large
----

We then wrote a function to check that a given point was actually in the array:

[source,haskell]
----

hasItemSingle :: [a] -> Int -> Bool
hasItemSingle array position = position >= 0 && (length array - 1) >= position

hasItem :: [[a]] -> Int -> Int -> Bool
hasItem array x y = hasItemSingle array x && hasItemSingle (array !! x) y
----

And our function to calculate the diagonals then used that:

[source,text]
----

> [findValue (1+z) (1+z) grid | z <- [0..19], hasItem grid (1+z) (1+z) ]
[49,31,23,51,3,67,20,97,45,3,24,44,52,26,32,40,4,5,48]
----

That function eventually ended up like this:

[source,haskell]
----

diagonalAt :: Int -> Int -> [[Int]] -> [Int]
diagonalAt x y grid = [findValue (x+z) (y+z) grid | z <- [0..(length grid)], hasItem grid (x + z) (y + z)]
----

The nice thing about taking this approach of only thinking about one function at a time is that you can actually make some progress since you only need to think about a small part of the problem as compared to the whole thing.

I still start off trying to solve a problem from the outside in but if I get stuck then I start looking for some simple functions that I can write to get started and help me work towards an overall solution.
Functional Programming: Handling the Options
Date: 2012-03-21 00:50:37

A couple of weeks ago https://twitter.com/#!/channingwalton/status/177662437905010688[Channing Walton tweeted the following]:

____
Every time you call get on an Option a kitten dies.
____

As Channing points out in the comments he was referring to unguarded calls to 'get' which would lead to an exception if the Option was empty, therefore pretty much defeating the point of using an Option in the first place!

We're using Dan Bodart's http://code.google.com/p/totallylazy/[totallylazy] library on the application I'm currently working on and in fact were calling 'get' on an Option so I wanted to see if we could get rid of it.

The code in question is used to work out a price based on a base value and an increment value, both of which are optional.

I've re-created some tests to show roughly how it works:

[source,java]
----

@Test
public void myOptionsExample() {
    Assert.assertThat(calculate(Option.<Double>none(), Option.<Double>none(), 20), is(Option.<Double>none()));
    Assert.assertThat(calculate(Option.<Double>none(), Option.some(12.0), 20), is(Option.some(240.0)));
    Assert.assertThat(calculate(Option.some(50.0), Option.some(12.0), 20), is(Option.some(290.0)));
    Assert.assertThat(calculate(Option.some(50.0), Option.<Double>none(), 20), is(Option.some(50.0)));
}
----

[source,java]
----

private Option<Double> calculate(Option<Double> base, final Option<Double> increment, final int multiplier) {
    if(base.isEmpty() && increment.isEmpty()) {
        return Option.none();
    }

    if(increment.isEmpty()){
        return base;
    }

    double baseBit = base.getOrElse(0.0);
    double incrementBit = increment.get() * multiplier;

    return Option.some(baseBit + incrementBit);
}
----

We can get rid of the 'increment.isEmpty()' line by making use of a 'getOrElse' a few lines later like so:

[source,java]
----

private Option<Double> calculate(Option<Double> base, final Option<Double> increment, final int multiplier) {
    if(base.isEmpty() && increment.isEmpty()) {
        return Option.none();
    }

    double baseBit = base.getOrElse(0.0);
    double incrementBit = increment.getOrElse(0.0) * multiplier;

    return Option.some(baseBit + incrementBit);
}
----

In Scala we can often make use of a for expression when combining Options but I'm not sure it would help in this case because we want to return a 'Some' if either of the Options has a value and only return a 'None' if they're both empty.

I can't figure out how to get rid of that opening if statement - I tried pushing the options into a list and then calling 'foldLeft' on that to sum up the components but we still have the problem of how to deal with returning a 'None' when the list is empty.

If you know how to simplify that code either using totallylazy or another language please let me know in the comments!
Haskell: Pattern matching data types with named fields
Date: 2012-03-31 22:49:18

One of my favourite things about coding in Haskell is that I often end up pattern matching against data types.

I've been playing around with modelling cars coming into and out from a car park and changing the state of the car park accordingly.

I started with these data type definitions:

[source,haskell]
----

data CarParkState = Available Bool Int Int | AlmostFull Bool Int Int | Full Bool Int deriving (Show)
data Action = Entering | Leaving deriving (Show)
data Sticker = Handicap | None deriving (Show)
----

which were used in the following function:

[source,haskell]
----

car_park_state :: CarParkState -> (Action, Sticker) -> CarParkState
car_park_state (Available _ 8 handicap) (Entering, None) = AlmostFull True 9 handicap
car_park_state (AlmostFull _ 11 handicap) (Entering, None) = Full True handicap
car_park_state (AlmostFull _ 9 handicap) (Leaving, None) = Available True 8 handicap

-- there are more states but I've cut them for brevity
----

This code isn't too bad but it's not obvious what the numbers '8', '11' and '9' are referring to without scrolling back up to the data type definition and checking.

By coincidence I came across a http://en.wikibooks.org/wiki/Haskell/More_on_datatypes[Haskell data types wiki page] which introduced me to the concept of named fields.

This means that rather than referring to the arguments of a data type by position we can give them a name instead:

[source,haskell]
----

data CarParkState = Available  { changed :: Bool, normalSpaces :: Int, handicapSpaces :: Int } |
                    AlmostFull { changed :: Bool, normalSpaces :: Int, handicapSpaces :: Int } |
                    Full       { changed :: Bool, handicapSpaces :: Int } deriving (Show)
----

The +++<cite>+++car_park_state+++</cite>+++ function now reads like this:

[source,haskell]
----

car_park_state :: CarParkState -> (Action, Sticker) -> CarParkState
car_park_state state@(Available  {normalSpaces=8})  (Entering, None) = AlmostFull True 9 (handicapSpaces state)
car_park_state state@(AlmostFull {normalSpaces=11}) (Entering, None) = Full True (handicapSpaces state)
car_park_state state@(AlmostFull {normalSpaces=9})  (Leaving, None)  = Available True 8 (handicapSpaces state)
----

There's more code in this solution than the original although to me +++<cite>+++car_park_state+++</cite>+++ is clearer since you can tell what it's doing just by glancing at the code.

I'm not sure which of these approaches is considered idiomatic Haskell but my feeling is that it's probably the former since it's more concise.

On the other hand, the value of named fields goes up as the total number of parameters in a data types increases since people will find it increasingly difficult to remember which parameter is in each position.