markhneedham/blog huggingface dataset:
IntelliJ: Find/Replace using regular expressions with capture groups
Date: 2012-03-30 06:21:00

Everyone now and then we end up having to write a bunch of mapping code and I quite like using IntelliJ's 'Replace' option to do it but always end up spending about 5 minutes trying to remember how to do capture groups so I thought I'd write it down this time.

Given the following text in our file:

[source,scala]
----

val mark = 0
val dave = 0
val john = 0
val alex = 0
----

Let's say we wanted to prefix each of those names with 'cool' and had decided not to use Column mode for whatever reason.

One way of doing that is to capture the names and then replace each of them with 'cool' appended on the beginning:

A (very hacky) find regex could be this:

[source,text]
----

\s([a-zA-Z]+)\s=
----

Where we capture all the letters in the variable name inside a group and then build our replacement string like so:

[source,text]
----

 cool$1 =
----

I always expect the capture group replacement syntax to be '\1', '\2' and so on but it actually uses a '$' instead.

Hopefully now I shall remember!
Choosing where to put the complexity
Date: 2012-03-06 01:17:30

On the current application I'm working on we need to make use of some data which comes from another system so we've created an import script which creates a copy of that data so that we can use it in our application.

In general we've been trying not to do too much manipulation of the data and keeping it close to the initial structure so that if something goes wrong with the import we can more easily trace the problem back to the original data source.

image::{{<siteurl>}}/uploads/2012/03/complexity.gif[Complexity,474]

While that approach has generally been fine we recently had a situation where the way the data was stored in the original database was quite de-normalised and recreating that structure made some of the code in the data access layer quite messy.

We therefore decided to change the import script to normalise the data, thereby simplifying our database access code.

For now the complexity trade off seems ok because we haven't had to change the schema that much so it'll still be reasonably easy to track the data back to the source.

We currently don't have any tests specifically around the data import because there's not very much logic going on but if the complexity increases and we start to see problems with the import process then we'll need to change that.

While we were having this conversation about where we should place the complexity I found it interesting that you could argue for each approach equally convincingly and neither really seemed definitively better - it was back to the world of http://www.markhneedham.com/blog/?s=trade+off[trade offs]!
Saving the values of dynamically populated dropdown on back button
Date: 2012-03-24 00:40:34

We wanted to be able to retain the value of a drop down menu that was being dynamically populated (via an AJAX call) when the user hit the back button but the AJAX request re-runs when we go hit back therefore losing our selection.

Our initial thinking was that we might be able to store the value of the dropdown in a hidden field and then restore it into the dropdown using jQuery on page load but that approach didn't work since hidden fields don't seem to retain their values when you hit back.

Another approach would be to update the URL with a new # value on each change of the dropdown and then parse the URL on page load to figure out which value should be selected.

That would probably work but it seemed a bit too complicated.

Eventually we realised that we could create an extra text box, 'hide' it by use of 'display:none' in our CSS and then copy the dropdown value into that every time it changed.

We ended up with code something like this to capture the selected value:

[source,javascript]
----

$("#dropdown-menu").live('change', function() {
  $("#hidden-field').val($(this).val());
});
----

And then we populated the dropdown with the saved value on the callback from the AJAX call:

[source,javascript]
----

$.ajax({
  url: 'url/to/populate_dropdown',
  success: function(data) {
    $("#dropdown-menu").html(data);
	$("#dropdown-menu").val($("#hidden-field").val());
  }
});
----

It seems to work pretty well and it's a simple technique as well so it worked in all the browsers that we tested it in.
Oracle Spatial: Querying by a point/latitude/longitude
Date: 2012-03-23 23:54:42

We're using Oracle Spatial on the application I'm working on and while most of the time any spatial queries we make are done from Java code we wanted to be able to run them directly from SQL as well to verify the code was working correctly.

We normally end up forgetting how to construct a query so I thought I'd document it.

Assuming we have a table +++<cite>+++table_with_shape+++</cite>+++ which has a column +++<cite>+++shape+++</cite>+++ which is a polygon, if we want to check whether a lat/long value interacts with that shape we can do that with the following query:

[source,text]
----

SELECT *
FROM table_with_shape tws
WHERE
SDO_ANYINTERACT(tws.shape, SDO_GEOMETRY(2001, 8307, SDO_POINT_TYPE(<LongValueHere>, <LatValueHere>, NULL), NULL, NULL) ) = 'TRUE'
----

The first parameter to +++<cite>+++SDO_GEOMETRY+++</cite>+++ http://docs.oracle.com/cd/B19306_01/appdev.102/b14255/sdo_objrelschema.htm#i1004087[defines the type of geometry] which in this case is a point.

The second parameter is the coordinate system which is 8307 since we're using the 'WGS 84 longitude/latitude' system.

The third parameter is our point and the rest of the parameters aren't interesting here so we pass null for them.
Mercurial: hg push to Google Code
Date: 2012-03-14 21:25:40

I wanted to make a change to add +++<cite>+++flatMap+++</cite>+++ to +++<cite>+++Option+++</cite>+++ in https://code.google.com/p/totallylazy/[totallylazy] so I had to https://code.google.com/r/mhneedham-totally-lazy/[clone the repository] and make the change.

I thought I'd then be able to just push the change using my Google user name and password but instead ended up with the following error:

[source,text]
----

➜  mhneedham-totally-lazy  hg push
pushing to https://m.h.needham@code.google.com/r/mhneedham-totally-lazy/
searching for changes
1 changesets found
http authorization required
realm: Google Code hg Repository
user: m.h.needham
password:
abort: HTTP Error 403: Forbidden
----

It turns out that you need to specifically set an option to use your Google account from the https://code.google.com/hosting/settings[settings page]:

image::{{<siteurl>}}/uploads/2012/03/googlecode.gif[Googlecode,600]

And then it works!
Haskell: Print friendly representation of an Array
Date: 2012-04-03 21:52:56

Quite frequently I play around with 2D arrays in Haskell but I've never quite worked out how to print them in a way that makes it easy to see the contents.

I'm using the array from the 'Data.Array' module because it seems to be easier to transform them into a new representation if I want to change a value in one of the cells.

The function to create one therefore looks like this:

[source,haskell]
----

import Data.Array

grid :: Int -> a -> Array(Int, Int) a
grid size value = array ((0,0),(size-1,size-1)) [((x,y),value) | x<-[0..size-1], y<-[0..size-1]]
----

Which we can use like this:

[source,haskell]
----

> grid 2 0
array ((0,0),(1,1)) [((0,0),0),((0,1),0),((1,0),0),((1,1),0)]
----

I wanted to get the output to read like this:

[source,text]
----

0 0
0 0
----

I initially tried to override the 'Show' implementation but wasn't very successful in trying to do that - I'm not sure whether that's actually possible but someone on the IRC channel suggested I should probably try and write my own function to print it out.

I ended up with the following:

[source,haskell]
----

printGrid :: Show a => Array (Int, Int) a -> IO [()]
printGrid grid = sequence $ map (putStrLn . textRepresentation) $ toSimpleArray grid

toSimpleArray :: Array (Int, Int) a -> [[a]]	
toSimpleArray grid = [[grid ! (x, y) | x<-[lowx..highx]] |  y<-[lowy..highy]]
	where ((lowx, lowy), (highx, highy)) =  bounds grid

textRepresentation :: Show a => [a] -> String
textRepresentation row = foldl (\acc y -> acc ++ (show y) ++ " ") "" row
----

The +++<cite>+++toSimpleArray+++</cite>+++ function converts the array back into a format which is easier to deal with. So for a simple array:

[source,haskell]
----

> toSimpleArray (grid 2 0)
[[0,0],[0,0]]
----

We then map over the new array and apply +++<cite>+++textRepresentation+++</cite>+++ over each row to get a text representation.

The +++<cite>+++textRepresentation+++</cite>+++ function works like this:

[source,haskell]
----

> textRepresentation [1, 2, 3]
"1 2 3 "
----

After that we map +++<cite>+++putStrLn+++</cite>+++ over the result which gives us a collection of IO monads.

Unfortunately that still doesn't print the array out so we need +++<cite>+++sequence+++</cite>+++ which I came across in the http://learnyouahaskell.com/input-and-output[Learn You A Haskell] tutorial:

____
sequence takes a list of I/O actions and returns an I/O actions that will perform those actions one after the other. The result contained in that I/O action will be a list of the results of all the I/O actions that were performed.
____

And eventually this is how we use +++<cite>+++printGrid+++</cite>+++:

[source,haskell]
----

> printGrid $ grid 2 0
0 0
0 0
[(),()]
----

There must be ways to simplify some of that code so if you can see any let me know in the comments!
neo4J: Searching for nodes by name
Date: 2012-04-20 07:10:57

As I mentioned in a post a few days ago I've been graphing connections between ThoughtWorks people using neo4j and wanted to build auto complete functionality so I can search for the names of people in the graph.

The solution I came up was to create a Lucene index with an entry for each node and a common property on each document in the index so that I'd be able to get all the index entries easily.

I created the index like this, using the https://github.com/maxdemarzi/neography/blob/master/lib/neography/rest.rb[neography gem]:

[source,ruby]
----

Neography::Rest.new.add_node_to_index("people", "type", "person", node)
----

I can then get all the names like this:

[source,ruby]
----

all_people = Neography::Rest.new.get_index("people", "type", "person").map { |n| n["data"]["name"] }
----

It seemed like there must be a better way to do this and http://twitter.com/#!/mesirii[Michael Hunger] was kind enough to show me a couple of cleaner solutions.

One way is to query the initial index rather than creating a new one:

[source,ruby]
----

all_people = Neography::Rest.new.find_node_index("people", "name:*").map { |n| n["data"]["name"] }
----

The 'find_node_index' method allows us to pass in a Lucene query which gets http://docs.neo4j.org/chunked/stable/rest-api-indexes.html#rest-api-find-node-by-query[executed via neo4j's REST API]. In this case we're using a wild card query on the 'name' property so it will return all documents.

This way of getting all the names seemed to be much more intensive than my other approach and when I ran it a few times in a row I was getting OutOfMemory errors. My graph only has a few thousand nodes in it so I'm not sure why that is yet.

I think it should be possible to query the Lucene index directly with the partial name but I was struggling to get spaces in the search term to encode correctly and was getting back no results.

Another approach is to use a cypher query to get a collection of all the nodes:

[source,ruby]
----

all_people = Neography::Rest.new.execute_query("start n=node(*) return n")["data"].map { |n| n[0]["data"]["name"] }
----

I imagine this approach wouldn't scale with graph size but for my graph it works just fine.
Haskell: Colour highlighting when writing to the shell
Date: 2012-04-29 00:01:07

I spent a few hours writing a simple front end on top of the http://www.markhneedham.com/blog/2012/04/25/algorithms-rabin-karp-in-haskell/[Rabin Karp algorithm] so that I could show the line of the first occurrence of a pattern in a piece of text on the shell.

I thought it would be quite cool if I could highlight the appropriate text on the line http://www.debian-administration.org/articles/460[like how grep does] when the '--color=auto' flag is supplied.

We can make use of http://en.wikipedia.org/wiki/ANSI_escape_code[ANSI escape codes] to do this.

We need to construct an escape sequence which starts with the 'ESC' character, represented by decimal value 27 or hexidecimal value 1b, followed by a left bracket ("["):

____
"\x1b[" or "\27["
____

In the second part of the sequence we can choose http://en.wikipedia.org/wiki/ANSI_escape_code#CSI_codes[a variety of different options]. In this case we use the following:

____
CSI n [;k] m SGR -- Select Graphic Rendition Sets SGR parameters, including text color. After CSI can be zero or more parameters separated with ;. With no parameters, CSI m is treated as CSI 0 m (reset / normal), which is typical of most of the ANSI escape sequences.
____

To change the colour of the text we need to set 'k' to one of the following values:

* 30 - Black
* 31 - Red
* 32 - Green
* 33 - Yellow
* 34 - Blue
* 35 - Magenta
* 36 - Cyan
* 37 - White

For example, if we want to print all future text in green then we can use the following escape sequence:

[source,haskell]
----

> putStrLn $ "\x1b[32m"
----

image::{{<siteurl>}}/uploads/2012/04/green.jpg[Green,600]

Ideally we should then reset the terminal which means passing a 'k' value of 0.

So if we just want to highlight one word we'd end up with this:

[source,haskell]
----

> putStrLn $ "\x1b[32m" ++ "highlight me" ++ "\x1b[0m" ++ " but not me"
----

image::{{<siteurl>}}/uploads/2012/04/green2.png[Green2,600]
Haskell: Int and Integer
Date: 2012-04-28 17:39:54

In my http://www.markhneedham.com/blog/2012/04/25/algorithms-rabin-karp-in-haskell/[last post about the Rabin Karp algorithm] I mentioned that I was having some problems when trying to write a hash function which closely matched its English description.

____
((rm-1 * ascii char) + (rm-2 * ascii char) + ... (r0 * ascii char)) % q where r = 256, q = 1920475943
____

This is my current version of the hash function:

[source,haskell]
----

hash = hash' globalR globalQ
hash' r q string m = foldl (\acc x -> (r * acc + ord x) `mod` q) 0 $ take m string
----

And my initial attempt to write the alternate version was this:

[source,haskell]
----

hash2 :: [Char] -> Int -> Int
hash2 = hash2' globalR  globalQ	
hash2' r q string m = (flip mod q . sum . map (\(pow, char) -> ord char *  (r ^ pow)) . zip [m-1, m-2..0]) string
----

Unfortunately this version breaks down as we try to create hashes for bigger lists of characters:

[source,text]
----

> hash "markusaere" 10
849698536
----

[source,text]
----

> hash2 "markusaere" 10
1239828806
----

We can see what's going on if we execute the core bit of the hash function in the REPL:

[source,haskell]
----

> map (\(pow, char) -> ord char * 256 ^ pow) $ zip [9, 8..0] "markusaere"
[0,0,8214565720323784704,30117822508040192,128642860449792,493921239040,1627389952,6619136,29184,101]
----

In this case the hash value for the first character is:

[source,haskell]
----

> 256 ^ 9 * ord 'm'
0
----

Be default if we do calculations on integers the 'http://en.wikibooks.org/wiki/Haskell/A_Miscellany_of_Types#Integers[Int]' type is used:

____
"Int" is the more common 32 or 64 bit integer. Implementations vary, although it is guaranteed to be at least 30 bits.
____

The maximum value a 64 bit Int can hold is 2^63^, a value that we're exceeding with the first two calculations in our list.

Since our calculation has exceeded the maximum value of a 64 bit integer we need to http://www.haskell.org/tutorial/numbers.html[coerce] our calculation to use the 'Integer' type which has arbitrary precision up to the limit of the machine's memory.

We can use the 'toInteger' function to do this:

[source,haskell]
----

> toInteger (256 ^ 9) * toInteger (ord 'm')
514737946632791328292864
----

We can change the core of the function to use 'toInteger' like so:

[source,haskell]
----

> map (\(pow, char) -> toInteger (ord char) * 256 ^ pow) $ zip [9, 8..0] "markusaere"
[514737946632791328292864,1789334175149826506752,8214565720323784704,30117822508040192,128642860449792,493921239040,1627389952,6619136,29184,101]
----

We can then change the hash function to read like this:

[source,haskell]
----

hash2 :: [Char] -> Int -> Int
hash2 value m = fromInteger $ hash2' (toInteger globalR) (toInteger globalQ) value m
	
hash2' :: Integer -> Integer -> [Char] -> Int -> Integer
hash2' r q string m = (sum $ map (\(pow, char) -> toInteger (ord char) *  toInteger (r ^ pow)) $ zip [m-1, m-2..0] string) `mod` q
----

When we call the function with the original parameters it now returns the correct answer:

[source,text]
----

> hash2 "markusaere" 10
849698536
----
Algorithms: Flood Fill in Haskell - Abstracting the common
Date: 2012-04-17 07:22:12

In the comments of http://www.markhneedham.com/blog/2012/04/07/algorithms-flood-fill-in-haskell/[my blog post describing the flood fill algorithm in Haskell] David Turner pointed out that the way I was passing the grid around was quite error prone.

[source,haskell]
----

floodFill :: Array (Int, Int) Colour ->  (Int, Int) -> Colour -> Colour -> Array (Int, Int) Colour
floodFill grid point@(x, y) target replacement =
  if((not $ inBounds grid point) ||  grid ! (x,y) /= target) then grid
  else
    gridNorth
    where grid' = replace grid point replacement
          gridEast = floodFill grid' (x+1, y) target replacement
          gridWest = floodFill gridEast (x-1, y) target replacement
          gridSouth = floodFill gridWest (x, y+1) target replacement
          gridNorth = floodFill gridSouth (x, y-1) target replacement
----

I actually did pass the wrong grid variable around while I was writing it and ended up quite confused as to why it wasn't working as I expected.

David's suggestion based on the above version of the algorithm was that I might want to use the state monad to thread state rather than explicitly passing it around like I am here.

I had a go at writing the function using the state monad but realised while doing so that rather than doing that I could fold over a cells neighbours rather than using the state monad to thread state.

I didn't see the 'neighbours' concept the first time I wrote it but David http://hpaste.org/66803[wrote a version of the algorithm] in which he introduced that.

This is the resulting function:

[source,haskell]
----

floodFill :: Array (Int, Int) Colour ->  (Int, Int) -> Colour -> Colour -> Array (Int, Int) Colour
floodFill grid point target replacement =
  if(target == replacement) then
    grid
  else	
    let gridWithSquareReplaced = if onGrid point then grid // [(point, replacement)] else grid
        validNeighbours = filter (onGrid `and` sameAsTarget) neighbours in
	
        foldl (\grid point -> floodFill grid point target replacement) gridWithSquareReplaced validNeighbours
		
        where neighbours = let (x,y) = point in [(x+1, y), (x-1, y), (x, y+1), (x, y-1)]
              sameAsTarget point = grid ! point == target
              onGrid = inRange $ bounds grid

and :: (a -> Bool) -> (a -> Bool) -> a -> Bool
and f g x = f x && g x
----

I initially get all of the neighbouring squares but then filter those out if they don't fit on the grid or aren't of the required colour. We can then use 'foldl' to iterate over each neighbour passing the grid along as the accumulator.

David also wrote a more succinct function to work out whether or not a value fits on the grid so I've stolen that as well!

I'm still not sure I totally understand when I should be using 'let' and 'where' but in this example the functions I've put in the 'where' section are helper functions whereas the values I defined in the 'let' section are domain concepts.

Whether that's the correct way to think of the two I'm not sure!
Algorithms: Flood Fill in Haskell
Date: 2012-04-07 00:25:34

http://en.wikipedia.org/wiki/Flood_fill[Flood fill] is an algorithm used to work out which nodes are connected to a certain node in a multi dimensional array. In this case we'll use a two dimensional array.

The idea is that we decide that we want to change the colour of one of the cells in the array and have its immediate neighbours who share its initial colour have their colour changed too i.e. the colour floods its way through the grid.

The algorithm is described on Wikipedia like so:

____
Flood-fill (node, target-color, replacement-color):

. If the color of node is not equal to target-color, return.
. Set the color of node to replacement-color.
. {blank}
 ** Perform Flood-fill (one step to the west of node, target-color, replacement-color).
 ** Perform Flood-fill (one step to the east of node, target-color, replacement-color).
 ** Perform Flood-fill (one step to the north of node, target-color, replacement-color).
 ** Perform Flood-fill (one step to the south of node, target-color, replacement-color).
. Return.+++<cite>+++replace+++</cite>++++++<cite>+++http://cvs.haskell.org/Hugs/pages/libraries/base/Data-Array.html#v%3Abounds[bounds]+++</cite>++++++<cite>+++toComplexArray+++</cite>+++https://github.com/mneedham/haskell/blob/master/PrintArray.hs[code for that is on github]+++<cite>+++printGrid+++</cite>+++http://www.markhneedham.com/blog/2012/04/03/haskell-print-friendly-representation-of-an-array/[in my last post]https://github.com/mneedham/haskell/blob/master/flood_fill.hs[my github haskell repository]
____
Just Observe
Date: 2012-04-09 22:45:07

One of the most common instincts of a developer when starting on a new team is to look at the way the application has been designed and find ways that it can be done differently.

Most often 'differently' means that a pattern used in a previous project will be favoured and while I think it's good to make use of experience that we've gained, we do miss out on some learning if we write every application the same way.

Therefore, unless I see a decision being made which is clearly going to lead to disaster later on, I now prefer to observe the approach being taken and understand its advantages and disadvantages before making a judgement.

A few years Jay Fields wrote a post titled 'http://blog.jayfields.com/2009/03/kill-your-darlings.html[Kill Your Darlings]' where he suggested the following:

____
You always gain by allowing someone to show you an alternative solution. If someone wants to solve a problem in a different way, there are several gains to be had. If their way is inferior, you have an opportunity to mentor a team-mate. If their way is equally elegant, you've gained another solution, or point of view that may be superior in the future. If their way is superior you learn something new and the codebase improves. In exchange for these gains you only need to give up time. Time is valuable, but it's also well spent on improving the ability of a team-mate or your personal ability.
____

He was talking in a slightly different context but it seems equally applicable here.

One example from the current project I'm working on is that we're not using an ORM tool but instead mapping from database queries to domain objects manually.

This is being done partly to get around the query optimisation problems we can end up with when using tools like Hibernate but it also means that we design our domain code properly and don't get tempted to make it match the database structure.

I initially thought that it was a strange decision to write code to do mapping when you have a tool to do it for you but it hasn't been as bad as I'd imagined although the data access code is pretty messy.

On the positive side we can directly measure the performance of each query, make the necessary tweaks, and then re-measure the performance much more easily this way.

The application is read only so we wouldn't have got to use the full 'CRUD' repertoire that Hibernate provides anyway.

It might not be the right choice for the next application that I work on but I'm glad I got the chance to see what an application written without an ORM looks like.

Just observe!