markhneedham/blog huggingface dataset:
Neo4j: Cypher - UNWIND vs FOREACH
Date: 2014-05-31 14:19:25

I've written a http://www.markhneedham.com/blog/2014/05/20/neo4j-2-0-creating-adjacency-matrices/[couple of^] http://www.markhneedham.com/blog/2014/05/25/neo4j-2-1-passing-around-node-ids-vs-unwind/[posts^] about the new http://docs.neo4j.org/chunked/milestone/query-unwind.html[UNWIND^] clause in Neo4j's cypher query language but I forgot about my favourite use of UNWIND, which is to get rid of some uses of http://docs.neo4j.org/chunked/stable/query-foreach.html[FOREACH^] from our queries.


Let's say we've http://www.markhneedham.com/blog/2014/04/19/neo4j-cypher-creating-a-time-tree-down-to-the-day/[created a timetree up front^] and now have a series of events coming in that we want to create in the database and attach to the appropriate part of the timetree.


Before UNWIND existed we might try to write the following query using FOREACH:



[source,cypher]
----

WITH [{name: "Event 1", timetree: {day: 1, month: 1, year: 2014}},
      {name: "Event 2", timetree: {day: 2, month: 1, year: 2014}}] AS events
FOREACH (event IN events |
  CREATE (e:Event {name: event.name})
  MATCH (year:Year {year: event.timetree.year }),
        (year)-[:HAS_MONTH]->(month {month: event.timetree.month }),
        (month)-[:HAS_DAY]->(day {day: event.timetree.day })
  CREATE (e)-[:HAPPENED_ON]->(day))
----

Unfortunately we can't use MATCH inside a FOREACH statement so we'll get the following error:



[source,cypher]
----

Invalid use of MATCH inside FOREACH (line 5, column 3)
"  MATCH (year:Year {year: event.timetree.year }), "
   ^
Neo.ClientError.Statement.InvalidSyntax
----

We can work around this by using MERGE instead in the knowledge that it's never going to create anything because the timetree already exists:



[source,cypher]
----

WITH [{name: "Event 1", timetree: {day: 1, month: 1, year: 2014}},
      {name: "Event 2", timetree: {day: 2, month: 1, year: 2014}}] AS events
FOREACH (event IN events |
  CREATE (e:Event {name: event.name})
  MERGE (year:Year {year: event.timetree.year })
  MERGE (year)-[:HAS_MONTH]->(month {month: event.timetree.month })
  MERGE (month)-[:HAS_DAY]->(day {day: event.timetree.day })
  CREATE (e)-[:HAPPENED_ON]->(day))
----

If we replace the FOREACH with UNWIND we'd get the following:



[source,cypher]
----

WITH [{name: "Event 1", timetree: {day: 1, month: 1, year: 2014}},
      {name: "Event 2", timetree: {day: 2, month: 1, year: 2014}}] AS events
UNWIND events AS event
CREATE (e:Event {name: event.name})
WITH e, event.timetree AS timetree
MATCH (year:Year {year: timetree.year }),
      (year)-[:HAS_MONTH]->(month {month: timetree.month }),
      (month)-[:HAS_DAY]->(day {day: timetree.day })
CREATE (e)-[:HAPPENED_ON]->(day)
----

Although the lines of code has slightly increased the query is now correct and we won't accidentally correct new parts of our time tree.

We could also pass on the event that we created to the next part of the query which wouldn't be the case when using FOREACH.
Thoughts on meetups
Date: 2014-05-31 19:50:26

I recently came across an http://blog.factual.com/clojure-office-hours[interesting blog post by Zach Tellman] in which he explains a new approach that he's been trialling at http://www.meetup.com/The-Bay-Area-Clojure-User-Group/events/181057342/[The Bay Area Clojure User Group].

Zach explains that a lecture based approach isn't necessarily the most effective way for people to learn and that half of the people attending the meetup are likely to be novices and would struggle to follow more advanced content.

He then goes on to explain an alternative approach:

____
We've been experimenting with a Clojure meetup modelled on a different academic tradition: *office hours*. At a university, students who have questions about the lecture content or coursework can visit the professor and have a one-on-one conversation. \... At the beginning of every meetup, we give everyone a name tag, and provide a whiteboard with two columns, "`teachers`" and "`students`". Attendees are encouraged to put their name and interests in both columns. From there, everyone can [\...] go in search of someone from the opposite column who shares their interests.
____

While running Neo4j meetups we've had similar observations and my colleagues https://twitter.com/darthvader42[Stefan] and https://twitter.com/neo4jfr[Cedric] http://www.meetup.com/graphdb-france/events/163480742/[actually ran a meetup in Paris a few months ago] which sounds very similar to Zach's 'office hours' style one.

However, we've also been experimenting with the idea that *one size doesn't need to fit all* by running different styles of meetups aimed at different people.

For example, we have:

* http://www.meetup.com/graphdb-london/events/184326312/[An introductory meetup] which aims to get people to the point where they can follow talks about more advanced topics.
* A http://www.meetup.com/graphdb-london/events/181678522/[more hands on session] for people who want to learn how to write queries in cypher, Neo4j's query language.
* An http://www.meetup.com/graphdb-london/events/181676642/[advanced session] for people who want to learn how to model a problem as a graph and import data into a graph.

I'm also thinking of running something similar to the http://londonclojurians.org/[Clojure Dojo] but focused on data and graphs where groups of people could work together and build an app.

I noticed that Nick Manning has been doing http://www.meetup.com/nycneo4j/events/152795422/[a] http://www.meetup.com/nycneo4j/events/149762532/[similar] http://www.meetup.com/nycneo4j/events/158975752/[thing] with the http://www.meetup.com/nycneo4j/[New York City Neo4j meetup] as well, which is cool.

I'd be interested in hearing about different / better approaches that other people have come across so if you know of any let me know in the comments.
Clojure: Destructuring group-by's output
Date: 2014-05-31 00:03:48

One of my favourite features of Clojure is that it allows you to destructure a data structure into values that are a bit easier to work with.

I often find myself referring to http://blog.jayfields.com/2010/07/clojure-destructuring.html[Jay Fields' article] which contains several examples showing the syntax and is a good starting point.

One recent use of destructuring I had was where I was working with a vector containing events like this:

[source,lisp]
----

user> (def events [{:name "e1" :timestamp 123} {:name "e2" :timestamp 456} {:name "e3" :timestamp 789}])
----

I wanted to split the events in two - those containing events with a timestamp greater than 123 and those less than or equal to 123.

After remembering that the function I wanted was +++<cite>+++http://clojuredocs.org/clojure_core/1.2.0/clojure.core/group-by[group-by]+++</cite>+++ and not +++<cite>+++http://clojuredocs.org/clojure_core/1.2.0/clojure.core/partition-by[partition-by]+++</cite>+++ (I always make that mistake!) I had the following:

[source,lisp]
----

user> (group-by #(> (->> % :timestamp) 123) events)
{false [{:name "e1", :timestamp 123}], true [{:name "e2", :timestamp 456} {:name "e3", :timestamp 789}]}
----

I wanted to get 2 vectors that I could pass to the web page and this is fairly easy with destructuring:

[source,lisp]
----

user> (let [{upcoming true past false} (group-by #(> (->> % :timestamp) 123) events)]
       (println upcoming) (println past))
[{:name e2, :timestamp 456} {:name e3, :timestamp 789}]
[{:name e1, :timestamp 123}]
nil
----

Simple!
Neo4j Meetup Coding Dojo Style
Date: 2014-05-31 22:55:33

A few weeks ago we ran a +++<cite>+++http://www.meetup.com/graphdb-london/events/179211972/[build your first Neo4j app]+++</cite>+++ meetup in the Neo4j London office during which we worked with the meta data around http://britishlibrary.typepad.co.uk/digital-scholarship/2013/12/a-million-first-steps.html[1 million images recently released into the public domain by the British Library].

Feedback from previous meetups had indicated that attendees wanted to practice modelling a domain from scratch and understand the options for importing said model into the database. This data set seemed perfect for this purpose.

We started off by scanning the data set and coming up with some potential questions we could ask of it and then the group split in two and came up with a graph model:

image::{{<siteurl>}}/uploads/2014/05/neo4j_dojo.png[Neo4j dojo,600]

Having spent 15 minutes working on that, one person from each group explained the process they'd gone through to all attendees.

Each group took a similar approach whereby they scanned a subset of the data, sketched out all the properties and then discussed whether or not something should be a node, relationship or property in a graph model.

We then spent a bit of time tweaking the model so we had one everyone was happy with.

We split into three groups to work on input. One group imported some of the data by generating cypher statements from Java, one imported data using http://book.py2neo.org/en/latest/[py2neo] and the last group imported data using the http://docs.neo4j.org/chunked/milestone/batchinsert.html[batch inserter].

You can have a look at the https://github.com/mneedham/neo4j-bl[github repository] to see what we got up and specifically the +++<cite>+++https://github.com/mneedham/neo4j-bl/tree/solution[solution]+++</cite>+++ branch to see the batch inserter code and the +++<cite>+++https://github.com/mneedham/neo4j-bl/tree/cypher-import[cypher-import]+++</cite>+++ branch for the cypher based approach.

The approach we used throughout the session is quite similar to a http://www.markhneedham.com/blog/2011/03/29/thoughtworks-university-coding-dojo-style/[Kake coding dojo] - something I first tried out when I was a trainer at ThoughtWorks University.

Although there were a few setup based things that could have been a bit slicker I think this format worked reasonably well and we'll use something similar at the http://www.meetup.com/graphdb-london/events/181676642/[next version in a couple of weeks time].

Feel free to come along if it sounds interesting!
Clojure: Create a directory
Date: 2014-05-24 00:12:56

I spent much longer than I should have done trying to work out how to create a directory in Clojure as part of an import script I'm working out so for my future self this is how you do it:

[source,lisp]
----

(.mkdir (java.io.File. "/path/to/dir/to/create"))
----

I'm creating a directory which contains today's date so I'd want something like 'members-2014-05-24' if I was running it today. The https://github.com/clj-time/clj-time[clj-time] library is very good for working with dates.

To create a folder containing today's date this is what we'd have:

[source,lisp]
----

(ns neo4j-meetup.core
  (:require [clj-time.format :as f]))

(def format-as-year-month-day (f/formatter "yyyy-MM-dd"))

(defn create-directory-for-today []
  (let [date (f/unparse format-as-year-month-day (t/now))]
    (.mkdir (java.io.File. (str "data/members-" date)))))
----

Initial code shamelessly stolen from https://gist.github.com/halfelf[Shu Wang's gist] so thanks to him as well!
Neo4j: Cypher - Rounding a float value to decimal places
Date: 2014-05-25 22:17:35

About 6 months ago http://www.jacqui.tk/[Jacqui Read] created a github issue explaining how she wanted to https://github.com/neo4j/neo4j/issues/1580[round a float value to a number of decimal places] but was unable to do so due to the +++<cite>+++http://docs.neo4j.org/chunked/stable/query-functions-mathematical.html#functions-round[round]+++</cite>+++ function not taking the appropriate parameter.

I found myself wanting to do the same thing last week where I initially had the following value:

[source,cypher]
----

RETURN toFloat("12.336666") AS value
----

I wanted to round that to 2 decimal places and https://twitter.com/wefreema[Wes] suggested multiplying the value before using ROUND and then dividing afterwards to achieve that.

For two decimal places we need to multiply and divide by 100:

[source,cypher]
----

WITH toFloat("12.336666") AS value
RETURN round(100 * value) / 100 AS value
----

[source,bash]
----

12.34
----

https://twitter.com/mesirii[Michael] suggested abstracting the number of decimal places like so:

[source,cypher]
----

WITH 2 as precision
WITH toFloat("12.336666") AS value, 10^precision AS factor
RETURN round(factor * value)/factor AS value
----

If we want to round to 4 decimal places we can easily change that:

[source,cypher]
----

WITH 4 as precision
WITH toFloat("12.336666") AS value, 10^precision AS factor
RETURN round(factor * value)/factor AS value
----

[source,bash]
----

12.3367
----

The code is http://gist.neo4j.org/?0e68a1c6922c04b53af0[available as a graph gist] too if you want to play around with it. And you may as well check out the https://github.com/neo4j-contrib/graphgist/wiki[other gists] while you're at it - enjoy!
Automating Skype's 'This message has been removed'
Date: 2014-02-20 23:16:34

One of the stranger features of Skype is that that it allows you to delete the contents of a message that you've already sent to someone - something I haven't seen on any other messaging system I've used.

For example if I wrote a message in Skype and wanted to edit it I would press the 'up' arrow:

image::{{<siteurl>}}/uploads/2014/02/2014-02-20_23-02-28.png[2014 02 20 23 02 28,564]

Once I've deleted the message I'd see this in the space where the message used to be:

image::{{<siteurl>}}/uploads/2014/02/2014-02-20_23-00-41.png[2014 02 20 23 00 41,600]

I almost certainly am too obsessed with this but I find it quite amusing when I see people posting and retracting messages so I wanted to see if it could be automated.

https://twitter.com/apcj[Alistair] showed me http://support.apple.com/kb/ht2488[Automator], a built in tool on the Mac for automating work flows.

Automator allows you to execute Applescript so we wrote the following code which selects the current chat in Skype, writes a message and then deletes it one character at a time:

[source,bash]
----

on run {input, parameters}
	tell application "Skype"
		activate
	end tell
	
	tell application "System Events"
		set message to "now you see me, now you don't"
		keystroke message
		keystroke return
		keystroke (ASCII character 30) --up arrow
		repeat length of message times
			keystroke (ASCII character 8) --backspace
		end repeat
		keystroke return
	end tell
	return input
end run
----

We wired up the Applescript via the Utilities > Run Applescript menu option in Automator:

image::{{<siteurl>}}/uploads/2014/02/2014-02-20_23-12-38.png[2014 02 20 23 12 38,380]

We can then go further and wire that up to a keyboard shortcut if we want by saving the workflow as a service in Automator but for my messing around purposes clicking the 'Run' button from Automator didn't seem too much of a hardship!
Neo4j: Cypher - Finding directors who acted in their own movie
Date: 2014-02-28 22:57:59

I've been doing quite a few Intro to Neo4j sessions recently and since it contains a lot of problems for the attendees to work on I get to see how first time users of Cypher actually use it.

A couple of hours in we want to write a query to find directors who acted in their own film based on the following model.

image::{{<siteurl>}}/uploads/2014/02/2014-02-28_22-40-02.png[2014 02 28 22 40 02,600]

A common answer is the following:

[source,cypher]
----

MATCH (a)-[:ACTED_IN]->(m)<-[:DIRECTED]-(d)
WHERE a.name = d.name
RETURN a
----

We're matching an actor 'a', finding the movie they acted in and then finding the director of that movie. We now have pairs of actors and directors which we filter down by comparing their 'name' property.

I haven't written SQL for a while but if my memory serves me correctly comparing properties or attributes in this way is quite a common way to test for equality.

In a graph we don't need to compare properties - what we actually want to check is if 'a' and 'd' are the same node:

[source,cypher]
----

MATCH (a)-[:ACTED_IN]->(m)<-[:DIRECTED]-(d)
WHERE a = d
RETURN a
----

We've simplifed the query a bit but we can actually go one better by binding the director to the same identifier as the actor like so:

[source,cypher]
----

MATCH (a)-[:ACTED_IN]->(m)<-[:DIRECTED]-(a)
RETURN a
----

So now we're matching an actor 'a', finding the movie they acted in and then finding the director if they happen to be the same person as 'a'.

The code is now much simpler and more revealing of its intent too.
Java 8: Sorting values in collections
Date: 2014-02-23 14:43:47

Having realised that Java 8 is http://openjdk.java.net/projects/jdk8/[due for its GA release within the next few weeks] I thought it was about time I had a look at it and over the last week have been reading http://pragprog.com/book/vsjava8/functional-programming-in-java[Venkat Subramaniam's book].

I'm up to chapter 3 which covers sorting a collection of people. The Person class is defined roughly like so:

[source,java]
----

static class Person {
    private String name;
    private int age;

    Person(String name, int age) {
        this.name = name;
        this.age = age;
    }

    @Override
    public String toString() {
        return String.format("Person{name='%s', age=%d}", name, age);
    }
}
----

In the first example we take a list of people and then sort them in ascending age order:

[source,java]
----

List<Person> people = Arrays.asList(new Person("Paul", 24), new Person("Mark", 30), new Person("Will", 28));
people.stream().sorted((p1, p2) -> p1.age - p2.age).forEach(System.out::println);
----

[source,text]
----

Person{name='Paul', age=24}
Person{name='Will', age=28}
Person{name='Mark', age=30}
----

If we were to write a function to do the same thing in Java 7 it'd look like this:

[source,java]
----

Collections.sort(people, new Comparator<Person>() {
    @Override
    public int compare(Person o1, Person o2) {
        return o1.age - o2.age;
    }
});

for (Person person : people) {
    System.out.println(person);
}
----

Java 8 has reduced the amount of code we have to write although it's still more complicated than what we could do in Ruby:

[source,ruby]
----

> people = [ {:name => "Paul", :age => 24}, {:name => "Mark", :age => 30}, {:name => "Will", :age => 28}]
> people.sort_by { |p| p[:age] }
=> [{:name=>"Paul", :age=>24}, {:name=>"Will", :age=>28}, {:name=>"Mark", :age=>30}]
----

A few pages later Venkat shows how you can get close to this by using the +++<cite>+++http://download.java.net/jdk8/docs/api/java/util/Comparator.html#comparing-java.util.function.Function-[Comparator#comparing]+++</cite>+++ function:

[source,java]
----

Function<Person, Integer> byAge = p -> p.age ;
people.stream().sorted(comparing(byAge)).forEach(System.out::println);
----

I thought I could make this simpler by inlining the 'byAge' lambda like this:

[source,java]
----

people.stream().sorted(comparing(p -> p.age)).forEach(System.out::println);
----

This seems to compile and run correctly although IntelliJ 13.0 suggests there is a 'http://youtrack.jetbrains.com/issue/IDEA-101788[cyclic inference]' problem. IntelliJ is happy if we explicitly cast the lambda like this:

[source,java]
----

people.stream().sorted(comparing((Function<Person, Integer>) p -> p.age)).forEach(System.out::println);
----

IntelliJ also seems happy if we explicitly type 'p' in the lambda, so I think I'll go with that for the moment:

[source,java]
----

people.stream().sorted(comparing((Person p) -> p.age)).forEach(System.out::println);
----
R: dplyr - Sum for group_by multiple columns
Date: 2014-11-11 00:17:32

Over the weekend I was playing around with dplyr and had the following data frame grouped by both columns:

[source,r]
----

> library(dplyr)

> data = data.frame(
    letter = sample(LETTERS, 50000, replace = TRUE),
    number = sample (1:10, 50000, replace = TRUE)
    )

> data %>% count(letter, number) %>% head(20)
Source: local data frame [20 x 3]
Groups: letter

   letter number   n
1       A      1 184
2       A      2 192
3       A      3 183
4       A      4 193
5       A      5 214
6       A      6 172
7       A      7 196
8       A      8 198
9       A      9 174
10      A     10 196
11      B      1 212
12      B      2 198
13      B      3 194
14      B      4 181
15      B      5 203
16      B      6 234
17      B      7 221
18      B      8 179
19      B      9 182
20      B     10 170
----

I wanted to add an extra column which would show what percentage of the values for that letter each number had.

If we wrote that code standalone we'd have the following:

[source,r]
----

> data %>% count(letter)
Source: local data frame [26 x 2]

   letter    n
1       A 1902
2       B 1974
3       C 1911
4       D 1948
5       E 1888
6       F 1975
7       G 1914
8       H 1886
9       I 1910
10      J 1924
11      K 1974
12      L 1891
13      M 1894
14      N 1946
15      O 1956
16      P 1934
17      Q 1865
18      R 1992
19      S 1946
20      T 1854
21      U 1919
22      V 1913
23      W 1928
24      X 1934
25      Y 1908
26      Z 1914
----

We can also get that value by summing up the individual (letter, number) pairs from the previous data frame. The +++<cite>+++ungroup+++</cite>+++ function allows us to do so:

[source,r]
----

> data %>% count(letter, number) %>% ungroup %>% group_by(letter) %>% mutate(sum.n = sum(n)) %>% head(20)
Source: local data frame [20 x 4]
Groups: letter

   letter number   n sum.n
1       A      1 184  1902
2       A      2 192  1902
3       A      3 183  1902
4       A      4 193  1902
5       A      5 214  1902
6       A      6 172  1902
7       A      7 196  1902
8       A      8 198  1902
9       A      9 174  1902
10      A     10 196  1902
11      B      1 212  1974
12      B      2 198  1974
13      B      3 194  1974
14      B      4 181  1974
15      B      5 203  1974
16      B      6 234  1974
17      B      7 221  1974
18      B      8 179  1974
19      B      9 182  1974
20      B     10 170  1974
----

Pretty neat!
R: dplyr - Select 'random' rows from a data frame
Date: 2014-11-26 00:01:12

Frequently I find myself wanting to take a sample of the rows in a data frame where just taking the head isn't enough.

Let's say we start with the following data frame:

[source,r]
----

data = data.frame(
    letter = sample(LETTERS, 50000, replace = TRUE),
    number = sample (1:10, 50000, replace = TRUE)
    )
----

And we'd like to sample 10 rows to see what it contains. We'll start by http://blog.revolutionanalytics.com/2009/02/how-to-choose-a-random-number-in-r.html[generating 10 random numbers] to represent row numbers using the +++<cite>+++runif+++</cite>+++ function:

[source,r]
----

> randomRows = sample(1:length(data[,1]), 10, replace=T)
> randomRows
 [1]  8723 18772  4964 36134 27467 31890 16313 12841 49214 15621
----

We can then pass that list of row numbers into dplyr's +++<cite>+++http://rpackages.ianhowson.com/cran/dplyr/man/slice.html[slice]+++</cite>+++ function like so:

[source,r]
----

> data %>% slice(randomRows)
   letter number
1       Z      4
2       F      1
3       Y      6
4       R      6
5       Y      4
6       V     10
7       R      6
8       D      6
9       J      7
10      E      2
----

If we're using that code throughout our code then we might want to pull out a function like so:

[source,r]
----

pickRandomRows = function(df, numberOfRows = 10) {
  df %>% slice(runif(numberOfRows,0, length(df[,1])))
}
----

And then call it like so:

[source,r]
----

> data %>% pickRandomRows()
   letter number
1       W      5
2       Y      3
3       E      6
4       Q      8
5       M      9
6       H      9
7       E     10
8       T      2
9       I      5
10      V      4

> data %>% pickRandomRows(7)
  letter number
1      V      7
2      N      4
3      W      1
4      N      8
5      G      7
6      V      1
7      N      7
----

== Update

https://twitter.com/tonkouts[Antonios] pointed out via email that we could just make use of the in-built +++<cite>+++sample_n+++</cite>+++ function which I didn't know about until now:

[source,r]
----

> data %>% sample_n(10)
      letter number
29771      U      1
48666      T     10
30635      A      1
34865      X      7
20140      A      3
41715      T     10
43786      E     10
18284      A      7
21406      S      8
35542      J      8
----