markhneedham/blog huggingface dataset:
Soulver: For all your random calculations
Date: 2014-03-30 14:48:41

I often find myself doing random calculations and I used to do so part manually and part using http://www.alfredapp.com/[Alfred]'s calculator until https://twitter.com/apcj[Alistair] pointed me at http://www.acqualia.com/soulver/[Soulver], a desktop/iPhone/iPad app, which is even better.

I thought I'd write some examples of calculations I use it for, partly so I'll remember the syntax in future!

_Calculating how much memory Neo4j memory mapping will take up_

[source,text]
----

800 mb + 2660mb + 6600mb + 9500mb + 40mb in GB = 19.6 GB
----

_How long would it take to cover 20,000 km at 100 km / day?_

[source,text]
----

20,000 km / 100 km/day in months = 6.57097681677241832481 months
----

_How long did an import of some data using the Neo4j shell take?_

[source,text]
----

4550855 ms in minutes = 75.84758333333333333333 minutes
----

_Bit shift 1 by 32 places_

[source,text]
----

1 << 32 = 4,294,967,296
----

_Translating into easier to digest units_

[source,text]
----

32381KB / second in MB per minute = 1,942.86 MB/minute
500,000 / 3 years in per hour = 19.01324310408685857874 per hour^2
----

_How long would it take to process a chunk of data?_

[source,text]
----

100 GB / (32381KB / second in MB per minute)  = 51.47051254336390681778 minutes
----

_Hexadecimal to base 10_

[source,text]
----

0x1111 = 4,369
1 + 16 + 16^2 + 16^3 = 4,369
----

I'm sure there's much more that you can do that I haven't figured out yet but even for these simple examples it saves me a bunch of time.
Remote profiling Neo4j using yourkit
Date: 2014-03-24 23:44:29

http://www.yourkit.com/[yourkit] is my favourite JVM profiling tool and whilst it's really easy to profile a local JVM process, sometimes I need to profile a process on a remote machine.

In that case we need to first have the remote JVM started up with a http://www.yourkit.com/docs/80/help/agent.jsp[yourkit agent] parameter passed as one of the args to the Java program.

Since I'm mostly working with Neo4j this means we need to add the following to +++<cite>+++conf/neo4j-wrapper.conf+++</cite>+++:

[source,text]
----

wrapper.java.additional=-agentpath:/Users/markhneedham/Downloads/YourKit_Java_Profiler_2013_build_13074.app/bin/mac/libyjpagent.jnilib=port=8888
----

If we run lsof with the Neo4j process ID we'll see that there's now a socket listening on port 8888:

[source,text]
----

java    4388 markhneedham   20u    IPv6 0x901df453b4e9a125       0t0      TCP *:8888 (LISTEN)
...
----

We can connect to that via the 'Monitor Remote Applications' section of yourkit:

image::{{<siteurl>}}/uploads/2014/03/2014-03-24_23-39-59.png[2014 03 24 23 39 59,600]

In this case I'm demonstrating how to connect to it on my laptop and am using localhost but usually we'd specify the remote machine's host name instead.

We also need to ensure that port 8888 is open on any firewalls we have in front of the machine.

The file we refer to in the 'agentpath' flag is a bit different depending on the operating system we're using. All the http://www.yourkit.com/docs/80/help/agent.jsp[details are on the yourkit website].
Neo4j: Cypher - Creating relationships between a collection of nodes / Invalid input '[':
Date: 2014-04-19 06:33:39

When working with graphs we'll frequently find ourselves wanting to create relationships between collections of nodes.

A common example of this would be creating a linked list of days so that we can quickly traverse across http://blog.neo4j.org/2012/02/modeling-multilevel-index-in-neoj4.html[a time tree]. Let's say we start with just 3 days:

[source,cypher]
----

MERGE (day1:Day {day:1 })
MERGE (day2:Day {day:2 })
MERGE (day3:Day {day:3 })
RETURN day1, day2, day3
----

And we want to create a 'NEXT' relationship between adjacent days:

[source,cypher]
----

(day1)-[:NEXT]->(day2)-[:NEXT]->(day3)
----

The most obvious way to do this would be to collect the days into an ordered collection and iterate over them using +++<cite>+++http://docs.neo4j.org/chunked/stable/query-foreach.html[FOREACH]+++</cite>+++, creating a relationship between adjacent nodes:

[source,cypher]
----

MATCH (day:Day)
WITH day
ORDER BY day.day
WITH COLLECT(day) AS days
FOREACH(i in RANGE(0, length(days)-2) |
  CREATE UNIQUE (days[i])-[:NEXT]->(days[i+1]))
----

Unfortunately this isn't valid syntax:

[source,text]
----

Invalid input '[': expected an identifier character, node labels, a property map, whitespace, ')' or a relationship pattern (line 6, column 32)
"            CREATE UNIQUE (days[i])-[:NEXT]->(days[i+1]))"
                                ^
----

It doesn't seem to like us using array indices where we specify the node identifier.

However, we can work around that by putting +++<cite>+++days[i]+++</cite>+++ and +++<cite>+++days[i+1]+++</cite>+++ into single item arrays and using nested +++<cite>+++FOREACH+++</cite>+++ loops on those, something http://www.markhneedham.com/blog/2013/11/29/neo4j-modelling-series-of-events/[Michael Hunger showed me last year] and I forgot all about!

[source,cypher]
----

MATCH (day:Day)
WITH day
ORDER BY day.day
WITH COLLECT(day) AS days
FOREACH(i in RANGE(0, length(days)-2) |
  FOREACH(day1 in [days[i]] |
    FOREACH(day2 in [days[i+1]] |
      CREATE UNIQUE (day1)-[:NEXT]->(day2))))
----

Now if we do a query to get back all the days we'll see they're connected:

image::{{<siteurl>}}/uploads/2014/04/2014-04-19_07-32-37.png[2014 04 19 07 32 37,366]
Neo4j 2.0.1: Cypher - Concatenating an empty collection / Type mismatch: expected Integer, Collection<Integer> or Collection<Collection<Integer>> but was Collection<Any>
Date: 2014-04-19 19:51:58

Last weekend I was playing around with some collections using Neo4j's Cypher query language and I wanted to concatenate two collections.

This was easy enough when both collections contained values\...

[source,cypher]
----

$ RETURN [1,2,3,4] + [5,6,7];
==> +---------------------+
==> | [1,2,3,4] + [5,6,7] |
==> +---------------------+
==> | [1,2,3,4,5,6,7]     |
==> +---------------------+
==> 1 row
----

\...but I ended up with the following exception when I tried to concatenate with an empty collection:

[source,cypher]
----

$ RETURN [1,2,3,4] + [];
==> SyntaxException: Type mismatch: expected Integer, Collection<Integer> or Collection<Collection<Integer>> but was Collection<Any> (line 1, column 20)
==> "RETURN [1,2,3,4] + []"
==>                     ^
----

I figured there was probably some strange type coercion going on for the empty collection and came up with the following work around using the +++<cite>+++http://docs.neo4j.org/chunked/stable/query-functions-collection.html#functions-range[RANGE]+++</cite>+++ function:

[source,cypher]
----

$ RETURN [1,2,3,4] + RANGE(0,-1);
==> +-------------------------+
==> | [1,2,3,4] + RANGE(0,-1) |
==> +-------------------------+
==> | [1,2,3,4]               |
==> +-------------------------+
==> 1 row
----

While writing this up I decided to check if it behaved the same way in the http://blog.neo4j.org/2014/04/neo4j-202-maintenance-release.html[recently released 2.0.2] and was pleasantly surprised to see that the work around is no longer necessary:

[source,cypher]
----

$ RETURN [1,2,3,4] + [];
==> +----------------+
==> | [1,2,3,4] + [] |
==> +----------------+
==> | [1,2,3,4]      |
==> +----------------+
==> 1 row
----

So if you're seeing the same issue get yourself upgraded!
Clojure: clj-time - Formatting a date / timestamp with day suffixes e.g. 1st, 2nd, 3rd
Date: 2014-04-26 07:50:46

I've been using the https://github.com/clj-time/clj-time[clj-time] library recently - a Clojure wrapper around http://joda-time.sourceforge.net/[Joda Time] - and one thing I wanted to do is format a date with day suffixes e.g. 1st, 2nd, 3rd.

I started with the following timestamp:

[source,text]
----

1309368600000
----

The first step was to convert that into a DateTime object like so:

[source,lisp]
----

user> (require '[clj-time.coerce :as c])
user> (c/from-long 1309368600000)
#<DateTime 2011-06-29T17:30:00.000Z>
----

I wanted to output that date in the following format:

[source,text]
----

29th June 2011
----

We can get quite close by using a https://github.com/clj-time/clj-time#clj-timeformat[custom time formatter]:

[source,lisp]
----

user> (require '[clj-time.format :as f])
nil
user> (f/unparse (f/formatter "d MMMM yyyy") (c/from-long 1309368600000))
"29 June 2011"
----

Unfortunately I couldn't find anywhere in the documentation explaining how to get the elusive 'th' or 'st' to print. I was hoping for something similar to PHP date formatting:

image::{{<siteurl>}}/uploads/2014/04/2014-04-26_08-38-39.png[2014 04 26 08 38 39,600]

Eventually I came across a Stack Overflow post about Joda Time suggesting that http://stackoverflow.com/questions/12950392/jodatime-format-date-with-1st-2nd-3rd-etc-day[you can't actually format a day in the way I was hoping to].

So I now have the following function to do it for me:

[source,lisp]
----

(defn day-suffix [day]
  (let [stripped-day (if (< day 20) day (mod day 10))]
    (cond (= stripped-day 1) "st"
          (= stripped-day 2) "nd"
          (= stripped-day 3) "rd"
          :else "th")))
----

and the code to get the date in my favoured format looks like this:

[source,lisp]
----

user> (def my-time (c/from-long 1309368600000))
#'user/my-time
user> (def day (read-string (f/unparse (f/formatter "d") my-time)))
#'user/day
user> (str day (day-suffix day) " " (f/unparse (f/formatter "MMMM yyyy") my-time))
"29th June 2011"
----

I'm assuming there's a better way but what is it?!
Jersey/Jax RS: Streaming JSON
Date: 2014-04-30 01:24:33

About a year ago I wrote a blog post showing how to http://www.markhneedham.com/blog/2013/07/08/jax-rs-streaming-a-response-using-streamingoutput/[stream a HTTP response using Jersey/Jax RS] and I recently wanted to do the same thing but this time using JSON.

A common pattern is to take our Java object and get a JSON string representation of that but that isn't the most efficient use of memory because we now have the Java object and a string representation.

This is particularly problematic if we need to return a lot of the data in a response.

By writing a little bit more code we can get our response to stream to the client as soon as some of it is ready rather than building the whole result and sending it all in one go:

[source,java]
----

@Path("/resource")
public class MadeUpResource
{
    private final ObjectMapper objectMapper;

    public MadeUpResource() {
        objectMapper = new ObjectMapper();
    }

    @GET
    @Produces(MediaType.APPLICATION_JSON)
    public Response loadHierarchy(@PathParam( "pkPerson" ) String pkPerson) {
        final Map<Integer, String> people  = new HashMap<>();
        people.put(1, "Michael");
        people.put(2, "Mark");

        StreamingOutput stream = new StreamingOutput() {
            @Override
            public void write(OutputStream os) throws IOException, WebApplicationException
            {
                JsonGenerator jg = objectMapper.getJsonFactory().createJsonGenerator( os, JsonEncoding.UTF8 );
                jg.writeStartArray();

                for ( Map.Entry<Integer, String> person : people.entrySet()  )
                {
                    jg.writeStartObject();
                    jg.writeFieldName( "id" );
                    jg.writeString( person.getKey().toString() );
                    jg.writeFieldName( "name" );
                    jg.writeString( person.getValue() );
                    jg.writeEndObject();
                }
                jg.writeEndArray();

                jg.flush();
                jg.close();
            }
        };


        return Response.ok().entity( stream ).type( MediaType.APPLICATION_JSON ).build()    ;
    }
}
----

If we run that this is the output we'd see:

[source,text]
----

[{"id":"1","name":"Michael"},{"id":"2","name":"Mark"}]
----

It's a simple example but hopefully it's easy to see how we could translate that if we wanted to stream more complex data.
Clojure: Paging meetup data using lazy sequences
Date: 2014-04-30 00:20:46

I've been playing around with the http://www.meetup.com/meetup_api/[meetup API] to do some analysis on the http://www.meetup.com/graphdb-london/[Neo4j London meetup] and one thing I wanted to do was download all the members of the group.

A feature of the meetup API is that each end point will only allow you to return a maximum of 200 records so I needed to make use of offsets and paging to retrieve everybody.

It seemed like a good chance to use some lazy sequences to keep track of the offsets and then stop making calls to the API once I wasn't retrieving any more results.

I wrote the following functions to take care of that bit:

[source,lisp]
----

(defn unchunk [s]
  (when (seq s)
    (lazy-seq
      (cons (first s)
            (unchunk (next s))))))

(defn offsets []
  (unchunk (range)))


(defn get-all [api-fn]
  (flatten
   (take-while seq
               (map #(api-fn {:perpage 200 :offset % :orderby "name"}) (offsets)))))
----

I previously wrote about the http://www.markhneedham.com/blog/2014/04/06/clojure-not-so-lazy-sequences-a-k-a-chunking-behaviour/[chunking behaviour of lazy collections] which meant that I ended up with a minimum of 32 calls to each URI which wasn't what I had in mind!

To get all the members in the group I wrote the following function which is passed to +++<cite>+++get-all+++</cite>+++:

[source,lisp]
----

(:require [clj-http.client :as client])

(defn members
  [{perpage :perpage offset :offset orderby :orderby}]
  (->> (client/get
        (str "https://api.meetup.com/2/members?page=" perpage
             "&offset=" offset
             "&orderby=" orderby
             "&group_urlname=" MEETUP_NAME
             "&key=" MEETUP_KEY)
        {:as :json})
       :body :results))
----

So to get all the members we'd do this:

[source,lisp]
----

(defn all-members []
  (get-all members))
----

I'm told that using lazy collections when side effects are involved is a bad idea - presumably because the calls to the API might never end - but since I only run it manually I can just kill the process if anything goes wrong.

I'd be interested in how others would go about solving this problem - https://github.com/clojure/core.async[core.async] was suggested but that seems to result in much more / more complicated code than this version.

The https://github.com/mneedham/neo4j-meetup/blob/master/src/neo4j_meetup/core.clj#L58[code is on github] if you want to take a look.
Clojure: Not so lazy sequences a.k.a chunking behaviour
Date: 2014-04-06 22:07:47

I've been playing with Clojure over the weekend and got caught out by the behaviour of lazy sequences due to chunking - something which was https://twitter.com/stilkov/status/452411800349605888[obvious] https://twitter.com/philandstuff/status/452412009074941952[to] https://twitter.com/Developerdave/status/452430573836193792[experienced] https://twitter.com/puredanger/status/452430894700048384[Clojurians] although not me.

I had something similar to the following bit of code which I expected to only evaluate the first item of the infinite sequence that the range function generates:

[source,lisp]
----

> (take 1 (map (fn [x] (println (str "printing..." x))) (range)))
(printing...0
printing...1
printing...2
printing...3
printing...4
printing...5
printing...6
printing...7
printing...8
printing...9
printing...10
printing...11
printing...12
printing...13
printing...14
printing...15
printing...16
printing...17
printing...18
printing...19
printing...20
printing...21
printing...22
printing...23
printing...24
printing...25
printing...26
printing...27
printing...28
printing...29
printing...30
printing...31
nil)
----

The reason this was annoying is because I wanted to shortcut the lazy sequence using +++<cite>+++http://clojuredocs.org/clojure_core/clojure.core/take-while[take-while]+++</cite>+++, much like the poster of http://stackoverflow.com/questions/3407876/how-do-i-avoid-clojures-chunking-behavior-for-lazy-seqs-that-i-want-to-short-ci[this StackOverflow question].

As I understand it when we have a lazy sequence the granularity of that laziness is 32 items at a time a.k.a one chunk, something that Michael Fogus http://blog.fogus.me/2010/01/22/de-chunkifying-sequences-in-clojure/[wrote about 4 years ago]. This was a bit surprising to me but it sounds like it makes sense for the majority of cases.

However, if we want to work around that behaviour we can wrap the lazy sequence in the following +++<cite>+++unchunk+++</cite>+++ function provided by Stuart Sierra:

[source,lisp]
----

(defn unchunk [s]
  (when (seq s)
    (lazy-seq
      (cons (first s)
            (unchunk (next s))))))
----

Now if we repeat our initial code we'll see it only prints once:

[source,lisp]
----

> (take 1 (map (fn [x] (println (str "printing..." x))) (unchunk (range))))
(printing...0
nil)
----
Neo4j: Cypher - Flatten a collection
Date: 2014-04-23 22:02:19

Every now and then in Cypher land we'll end up with a collection of arrays, often created via the http://docs.neo4j.org/chunked/stable/query-aggregation.html#aggregation-collect[COLLECT] function, that we want to squash down into one array.

For example let's say we have the following array of arrays\...

[source,cypher]
----

$ RETURN [[1,2,3], [4,5,6], [7,8,9]] AS result;
==> +---------------------------+
==> | result                    |
==> +---------------------------+
==> | [[1,2,3],[4,5,6],[7,8,9]] |
==> +---------------------------+
==> 1 row
----

\...and we want to return the array +++<cite>+++[1,2,3,4,5,6,7,8,9]+++</cite>+++.

Many programming languages have a 'flatten' function and although cypher doesn't we can make our own by using the +++<cite>+++http://docs.neo4j.org/chunked/stable/query-functions-collection.html#functions-reduce[REDUCE]+++</cite>+++ function:

[source,cypher]
----

$ WITH  [[1,2,3], [4,5,6], [7,8,9]] AS result
  RETURN REDUCE(output = [], r IN result | output + r) AS flat;
==> +---------------------+
==> | flat                |
==> +---------------------+
==> | [1,2,3,4,5,6,7,8,9] |
==> +---------------------+
==> 1 row
----

Here we're passing the array 'output' over the collection and adding the individual arrays (+++<cite>+++[1,2,3]+++</cite>+++, +++<cite>+++[4,5,6]+++</cite>+++ and +++<cite>+++[7,8,9]+++</cite>+++) to that array as we iterate over the collection.

If we're working with numbers in Neo4j 2.0.1 we'll get this type exception with this version of the code:

[source,text]
----

==> SyntaxException: Type mismatch: expected Any, Collection<Any> or Collection<Collection<Any>> but was Integer (line 1, column 148)
----

We can easily work around that by coercing the type of 'output' like so:

[source,cypher]
----

WITH  [[1,2,3], [4,5,6], [7,8,9]] AS result
RETURN REDUCE(output = range(0,-1), r IN result | output + r);
----

Of course this is quite a simple example but we can handle more complicated scenarios as well by using nested calls to REDUCE. For example let's say we wanted to completely flatten this array:

[source,cypher]
----

$ RETURN [[1,2,3], [4], [5, [6, 7]], [8,9]] AS result;
==> +-------------------------------+
==> | result                        |
==> +-------------------------------+
==> | [[1,2,3],[4],[5,[6,7]],[8,9]] |
==> +-------------------------------+
==> 1 row
----

We could write the following cypher code:

[source,cypher]
----

$ WITH  [[1,2,3], [4], [5, [6, 7]], [8,9]] AS result
  RETURN REDUCE(output = [], r IN result | output + REDUCE(innerOutput = [], innerR in r | innerOutput + innerR)) AS flat;
==> +---------------------+
==> | flat                |
==> +---------------------+
==> | [1,2,3,4,5,6,7,8,9] |
==> +---------------------+
==> 1 row
----

Here we have an outer REDUCE function which iterates over +++<cite>+++[1,2,3]+++</cite>+++, +++<cite>+++[4]+++</cite>+++, +++<cite>+++[5, [6,7]]+++</cite>+++ and +++<cite>+++[8,9]+++</cite>+++ and then an inner REDUCE function which iterates over those individual arrays.

If we had more nesting then we could just introduce another level of nesting!
Neo4j 2.0.0: Query not prepared correctly / Type mismatch: expected Map
Date: 2014-04-13 17:40:05

I was playing around with Neo4j's Cypher last weekend and found myself accidentally running some queries against an earlier version of the Neo4j 2.0 series (2.0.0).

My first query started with a map and I wanted to create a person from an identifier inside the map:

[source,cypher]
----

WITH {person: {id: 1}} AS params
MERGE (p:Person {id: params.person.id})
RETURN p
----

When I ran the query I got this error:

[source,text]
----

==> SyntaxException: Type mismatch: expected Map but was Boolean, Number, String or Collection<Any> (line 1, column 62)
==> "WITH {person: {id: 1}} AS params MERGE (p:Person {id: params.person.id}) RETURN p"
----

If we try the same query in 2.0.1 it works as we'd expect:

[source,text]
----

==> +---------------+
==> | p             |
==> +---------------+
==> | Node[1]{id:} |
==> +---------------+
==> 1 row
==> Nodes created: 1
==> Properties set: 1
==> Labels added: 1
==> 47 ms
----

My next query was the following which links topics of interest to a person:

[source,cypher]
----

WITH {topics: [{name: "Java"}, {name: "Neo4j"}]} AS params
MERGE (p:Person {id: 2})
FOREACH(t IN params.topics |
  MERGE (topic:Topic {name: t.name})
  MERGE (p)-[:INTERESTED_IN]->(topic)
)
RETURN p
----

In 2.0.0 that query fails like so:

[source,text]
----

==> InternalException: Query not prepared correctly!
----

but if we try it in 2.0.1 we'll see that it works as well:

[source,text]
----

==> +---------------+
==> | p             |
==> +---------------+
==> | Node[4]{id:2} |
==> +---------------+
==> 1 row
==> Nodes created: 1
==> Relationships created: 2
==> Properties set: 1
==> Labels added: 1
==> 53 ms
----

So if you're seeing either of those errors then http://www.neo4j.org/download[get yourself upgraded] to 2.0.1 as well!
Neo4j: Cypher - Neo.ClientError.Statement.ParameterMissing and neo4j-shell
Date: 2014-05-31 12:44:10

Every now and then I get sent Neo4j cypher queries to look at and more often than not they're http://docs.neo4j.org/chunked/stable/cypher-parameters.html[parameterised] which means you can't easily run them in the http://neo4j.com/blog/neo4j-2-0-0-m06-introducing-neo4js-browser/[Neo4j browser].

For example let's say we have a database which has a user called 'Mark':

[source,cypher]
----

CREATE (u:User {name: "Mark"})
----

Now we write a query to find 'Mark' with the name parameterised so we can easily search for a different user in future:

[source,cypher]
----

MATCH (u:User {name: {name}}) RETURN u
----

If we run that query in the Neo4j browser we'll get this error:

[source,bash]
----

Expected a parameter named name
Neo.ClientError.Statement.ParameterMissing
----

If we try that in neo4j-shell we'll get the same exception to start with:

[source,cypher]
----

$ MATCH (u:User {name: {name}}) RETURN u;
ParameterNotFoundException: Expected a parameter named name
----

However, as https://twitter.com/mesirii[Michael] pointed out to me, the neat thing about neo4j-shell is that we can define parameters by using the +++<cite>+++export+++</cite>+++ command:

[source,cypher]
----

$ export name="Mark"
$ MATCH (u:User {name: {name}}) RETURN u;
+-------------------------+
| u                       |
+-------------------------+
| Node[1923]{name:"Mark"} |
+-------------------------+
1 row
----

+++<cite>+++export+++</cite>+++ is a bit sensitive to spaces so it's best to keep them to a minimum. e.g. the following tries to create the variable 'name ' which is invalid:

[source,bash]
----

$ export name = "Mark"
name  is no valid variable name. May only contain alphanumeric characters and underscores.
----

The variables we create in the shell don't have to only be primitives. We can create maps too:

[source,bash]
----

$ export params={ name: "Mark" }
$ MATCH (u:User {name: {params}.name}) RETURN u;
+-------------------------+
| u                       |
+-------------------------+
| Node[1923]{name:"Mark"} |
+-------------------------+
1 row
----

A simple tip but one that saves me from having to rewrite queries all the time!