markhneedham/blog huggingface dataset:
Deliberate Practice: Building confidence vs practicing
Date: 2015-04-30 07:48:38

A few weeks ago I wrote about the http://www.markhneedham.com/blog/2015/04/07/neo4j-the-learning-to-cycle-dependency-graph/[learning to cycle dependency graph] which described some of the skills required to become proficient at riding a bike.

image::{{<siteurl>}}/uploads/2015/04/IMG_20150430_073120.jpg[IMG 20150430 073120,300]

While we've been practicing various skills/sub skills I've often found myself saying the following:

____
if it's not hard you're not practicing me, April 2015
____

i.e. you should find the skill you're currently practicing difficult otherwise you're not stretching yourself and therefore aren't getting better.

For example, in cycling you could be very comfortable riding with both hands on the handle bars and find using one hand a struggle. However, if you don't practice that you won't be able to indicate and turn corners.

This ties in with all my reading about deliberate practice which suggests that the type of exercises you do while deliberately practicing aren't intended to be fun and are meant to http://www.markhneedham.com/blog/2015/04/25/deliberate-practice-watching-yourself-fail/[expose your lack of knowledge].

In an ideal world we would spend all our time practicing these challenging skills but in reality there's some part of us that wants to feel that we're actually improving by spending some of the time doing things that we're good at. Doing things you're not good at is a bit of a slog as well so we might find that we have less motivation for this type of thing.

We therefore need to find *a balance between doing challenging exercises and having fun building something* or writing code that we already know how to do. I've found the best way to do this is to combine the two types of work into mini projects which contain some tasks that we're already good at and some that require us to struggle.

For me this might involved cleaning up and importing a data set into Neo4j, which I'm comfortable with, and combining that with something else that I want to learn.

For example in the middle of last year I did some http://www.markhneedham.com/blog/2014/05/31/neo4jr-analysing-london-nosql-meetup-membership/[meetup analysis] which involved creating a Neo4j graph of London's NoSQL meetups and learning a bit about R, dplyr and linear regression along the way.

In January I built a http://www.markhneedham.com/blog/2015/02/19/pythons-pandas-vs-neo4js-cypher-exploring-popular-phrases-in-how-i-met-your-mother-transcripts/[How I met your mother graph] and then spent a few weeks learning various algorithms for extracting topics from free text to give even more ways to explore the dataset.

Most recently I've been practicing exercises from http://www.greenteapress.com/thinkbayes/[Think Bayes] and while it's good practice I think I'd probably spend more time doing it if I linked it into a mini project with something I'm already comfortable with.

I'll go off and have a think what that should be!
Coding: Visualising a bitmap
Date: 2015-05-03 00:19:51

Over the last month or so I've spent some time each day reading a new part of the Neo4j code base to get more familiar with it, and one of my favourite classes is the https://github.com/neo4j/neo4j/blob/2.3/community/kernel/src/main/java/org/neo4j/kernel/impl/util/Bits.java[Bits] class which does all things low level on the wire and to disk.

In particular I like its toString method which returns a binary representation of the values that we're storing in bytes, ints and longs.

I thought it'd be a fun exercise to try and write my own function which takes in a 32 bit map and returns a string containing a 1 or 0 depending if a bit is set or not.

The key insight is that we need to iterate down from the highest order bit and then create a bit mask of that value and do a bitwise and with the full bitmap. If the result of that calculation is 0 then the bit isn't set, otherwise it is.

For example, to check if the highest order bit (index 31) was set our bit mask would have the 32nd bit set and all of the others 0'd out.

[source,java]
----

java> (1 << 31) & 0x80000000
java.lang.Integer res5 = -2147483648
----

If we wanted to check if lowest order bit was set then we'd run this computation instead:

[source,java]
----

java> (1 << 0) & 0x00000001
java.lang.Integer res7 = 0

java> (1 << 0) & 0x00000001
java.lang.Integer res8 = 1
----

Now let's put that into a function which checks all 32 bits of the bitmap rather than just the ones we define:

[source,java]
----

private String  asString( int bitmap )
{
    StringBuilder sb = new StringBuilder();
    sb.append( "[" );
    for ( int i = Integer.SIZE - 1; i >= 0; i-- )
    {
        int bitMask = 1 << i;
        boolean bitIsSet = (bitmap & bitMask) != 0;
        sb.append( bitIsSet ? "1" : "0" );

        if ( i > 0 &&  i % 8 == 0 )
        {
            sb.append( "," );
        }
    }
    sb.append( "]" );
    return sb.toString();
}
----

And a quick test to check it works:

[source,java]
----

@Test
public void shouldInspectBits()
{
    System.out.println(asString( 0x00000001 ));
    // [00000000,00000000,00000000,00000001]

    System.out.println(asString( 0x80000000 ));
    // [10000000,00000000,00000000,00000000]

    System.out.println(asString( 0xA0 ));
    // [00000000,00000000,00000000,10100000]

    System.out.println(asString( 0xFFFFFFFF ));
    // [11111111,11111111,11111111,11111111]
}
----

Neat!
Neo4j: LOAD CSV - java.io.InputStreamReader there's a field starting with a quote and whereas it ends that quote there seems  to be character in that field after that ending quote. That isn't supported.
Date: 2015-05-04 09:56:22

I recently came across the http://www.dtic.upf.edu/~ocelma/MusicRecommendationDataset/lastfm-360K.html[last.fm dataset] via http://www.benfrederickson.com/distance-metrics/[Ben Frederickson's blog] and thought it'd be an interesting one to load into Neo4j and explore.

I started with a simple query to parse the CSV file and count the number of rows:

[source,cypher]
----

LOAD CSV FROM "file:///Users/markneedham/projects/neo4j-recommendations/lastfm-dataset-360K/usersha1-artmbid-artname-plays.tsv"
AS row FIELDTERMINATOR  "\t"
return COUNT(*)

At java.io.InputStreamReader@4d307fda:6484 there's a field starting with a quote and whereas it ends that quote there seems  to be character in that field after that ending quote. That isn't supported. This is what I read: 'weird al"'
----

This blows up because (as the message says) we've got a field which uses double quotes but then has other characters either side of the quotes.

A quick search through the file reveals one of the troublesome lines:

[source,bash]
----

$ grep "\"weird" lastfm-dataset-360K/usersha1-artmbid-artname-plays.tsv  | head -n 1
0015371426d2cbef354b2f680340de38d0ebd2f0	7746d775-9550-4360-b8d5-c37bd448ce01	"weird al" yankovic	4099
----

I ran a file containing only that line through http://csvlint.io/validation/55473958637376058f000000[CSV Lint] to see what it thought and indeed it is invalid:

image::{{<siteurl>}}/uploads/2015/05/2015-05-04_10-50-43.png[2015 05 04 10 50 43,597]

Let's clean up our file to use single quotes instead of double quotes and try the query again:

[source,bash]
----

$ tr "\"" "'" < lastfm-dataset-360K/usersha1-artmbid-artname-plays.tsv > lastfm-dataset-360K/clean.tsv
----

[source,cypher]
----

LOAD CSV FROM "file:///Users/markneedham/projects/neo4j-recommendations/lastfm-dataset-360K/clean.tsv" as row FIELDTERMINATOR  "\t"
return COUNT(*)

17559530
----

And we're back in business! Interestingly Python's CSV reader chooses to strip out the double quotes rather than throw an exception:

[source,python]
----

import csv
with open("smallWeird.tsv", "r") as file:
    reader = csv.reader(file, delimiter="\t")

    for row in reader:
        print row
----

[source,bash]
----

$ python explore.py
['0015371426d2cbef354b2f680340de38d0ebd2f0', '7746d775-9550-4360-b8d5-c37bd448ce01', 'weird al yankovic', '4099']
----

I prefer LOAD CSV's approach but it's an interesting trade off I hadn't considred before.
Neo4j: Finding all shortest paths
Date: 2015-05-19 22:45:48

One of the Cypher language features we show in Neo4j training courses is the http://neo4j.com/docs/milestone/query-match.html#_shortest_path[shortest path function] which allows you to find the shortest path in terms of number of relationships between two nodes.

Using the movie graph, which you can import via the ':play movies' command in the browser, we'll first create a 'KNOWS' relationship between any people that have appeared in the same movie:

[source,cypher]
----

MATCH (p1:Person)-[:ACTED_IN]->()<-[:ACTED_IN]-(p2:Person)
MERGE (p1)-[:KNOWS]-(p2)
----

Now that we've got that relationship we can easily find the shortest path between two people, say Tom Cruise and Tom Hanks:

[source,cypher]
----

MATCH (p1:Person {name: "Tom Hanks"}), (p2:Person {name: "Tom Cruise"}),
      path = shortestpath((p1)-[:KNOWS*]-(p2))
RETURN path
----

image::{{<siteurl>}}/uploads/2015/05/graph-18.png[Graph  18,300]

That works pretty well but what if we want to find the longest shortest path between any two people in the graph? We can calculate it like this:

[source,cypher]
----

MATCH (p1:Person), (p2:Person),
      path = shortestpath((p1)-[:KNOWS*]-(p2))
RETURN path
ORDER BY LENGTH(path) DESC
LIMIT 1
----

image::{{<siteurl>}}/uploads/2015/05/graph-19.png[Graph  19,500]

So that's 6 hops which is actually the http://en.wikipedia.org/wiki/Six_Degrees_of_Kevin_Bacon[Bacon number] - I expect we'd probably see a smaller maximum value if we imported all the movies.

And to round off the post what if we want to find the longest shortest path between the 10 people who acted in the most movies? We might start out with the following query which seems like it should do the job:

[source,cypher]
----

MATCH (p1:Person)-[:ACTED_IN]->()

WITH p1, COUNT(*) AS appearances
ORDER BY appearances DESC
LIMIT 10

WITH p1 AS p1, p1 AS p2
MATCH path = shortestpath((p1)-[:KNOWS*]-(p2))
RETURN path
ORDER BY LENGTH(path) DESC
LIMIT 1
----

*Unfortunately if we run that query we get no rows returned because 'p1' and 'p2' always refer to the same node.*

Instead we can calculate the shortest path between our hardest working people by creating a cross product using COLLECT and UNWIND:

[source,cypher]
----

MATCH (p1:Person)-[:ACTED_IN]->()

WITH p1, COUNT(*) AS appearances
ORDER BY appearances DESC
LIMIT 10

WITH COLLECT(p1) AS ps
UNWIND ps AS p1 UNWIND ps AS p2
MATCH path = shortestpath((p1)-[:KNOWS*]-(p2))
RETURN path
ORDER BY LENGTH(path) DESC
LIMIT 1
----

image::{{<siteurl>}}/uploads/2015/05/graph-20.png[Graph  20,400]

That's all for now!
Python: CSV writing - TypeError: 'builtin_function_or_method' object has no attribute '*getitem*'
Date: 2015-05-31 22:33:54

When I'm working in Python I often find myself writing to CSV files using the https://docs.python.org/2/library/csv.html[in built library] and every now and then make a mistake when calling writerow:

[source,python]
----

import csv
writer = csv.writer(file, delimiter=",")
writer.writerow["player", "team"]
----

This results in the following error message:

[source,text]
----

TypeError: 'builtin_function_or_method' object has no attribute '__getitem__'
----

The error message is a bit weird at first but it's basically saying that I've tried to do an http://stackoverflow.com/questions/13075632/typeerror-builtin-function-or-method-object-has-no-attribute-getitem[associative lookup on an object which doesn't support that operation].

The resolution is simply to include the appropriate parentheses instead of leaving them out!

[source,python]
----

writer.writerow(["player", "team"])
----

This one's for future Mark.
Python: Joining multiple generators/iterators
Date: 2015-05-24 23:51:25

In my previous blog post I described how I'd http://www.markhneedham.com/blog/2015/05/23/python-refactoring-to-iterator/[refactored some scraping code I've been working on to use iterators] and ended up with a function which returned a generator containing all the events for one BBC live text match:

[source,python]
----

match_id = "32683310"
events = extract_events("data/raw/%s" % (match_id))

>>> print type(events)
<type 'generator'>
----

The next thing I wanted to do is get the events for multiple matches which meant I needed to glue together multiple generators into one big generator.

itertools' +++<cite>+++https://docs.python.org/2/library/itertools.html#itertools.chain[chain]+++</cite>+++ function http://stackoverflow.com/questions/3211041/how-to-join-two-generators-in-python[does exactly what we want]:

____
itertools.chain(*iterables) Make an iterator that returns elements from the first iterable until it is exhausted, then proceeds to the next iterable, until all of the iterables are exhausted. Used for treating consecutive sequences as a single sequence.
____

First let's try it out on a collection of range generators:

[source,python]
----

import itertools
gens = [(n*2 for n in range(0, 3)), (n*2 for n in range(4,7))]
>>> gens
[<generator object <genexpr> at 0x10ff3b140>, <generator object <genexpr> at 0x10ff7d870>]

output = itertools.chain()
for gen in gens:
  output = itertools.chain(output, gen)
----

Now if we iterate through 'output' we'd expect to see the multiples of 2 up to and including 12:

[source,python]
----

>>> for item in output:
...   print item
...
0
2
4
8
10
12
----

Exactly as we expected! Our scraping code looks like this once we plug the chaining in:

[source,python]
----

matches = ["32683310", "32683303", "32384894", "31816155"]

raw_events = itertools.chain()
for match_id in matches:
    raw_events = itertools.chain(raw_events, extract_events("data/raw/%s" % (match_id)))
----

'raw_events' now contains a single generator that we can iterate through and process the events for all matches.
R/ggplot: Controlling X axis order
Date: 2015-02-27 00:49:54

As part of a talk I gave at the Neo4j London meetup earlier this week I wanted to show how you could build a simple chart showing the number of friends that different actors had using the ggplot library.

I started out with the following code:

[source,r]
----

df = read.csv("/tmp/friends.csv")
top = df %>% head(20)

ggplot(aes(x = p.name, y = colleagues), data = top) +
  geom_bar(fill = "dark blue", stat = "identity")
----

The https://gist.github.com/mneedham/070d82a1e2aa031f42e6[friends CSV file is available as a gist] if you want to reproduce the chart. This is how the chart renders:

image::{{<siteurl>}}/uploads/2015/02/2015-02-27_00-41-08.png[2015 02 27 00 41 08,600]

It's in a fairly arbitrary order when it would be quite cool if we could get the most popular people to show from left to right.

I had the people's names in the correct order in the data frame but annoyingly it was then sorting them into alphabetical order. Luckily I came across the http://stackoverflow.com/questions/3253641/how-to-change-the-order-of-a-discrete-x-scale-in-ggplot[by using the +++<cite>+++scale_x_discrete+++</cite>+++ function] which does exactly what I needed.

If we pass in the list of names to that function we get the chart we desire:

[source,r]
----

ggplot(aes(x = p.name, y = colleagues), data = top) +
  geom_bar(fill = "dark blue", stat = "identity") +
  scale_x_discrete(limits= top$p.name)
----

image::{{<siteurl>}}/uploads/2015/02/2015-02-27_00-45-01.png[2015 02 27 00 45 01,600]
Python/pandas: Column value in list (ValueError: The truth value of a Series is ambiguous.)
Date: 2015-02-16 21:39:16

I've been using Python's http://pandas.pydata.org/[pandas] library while exploring some CSV files and although for the most part I've found it intuitive to use, I had trouble filtering a data frame based on checking whether a column value was in a list.

A subset of one of the CSV files I've been working with looks like this:

[source,python]
----

$ cat foo.csv
"Foo"
1
2
3
4
5
6
7
8
9
10
----

Loading it into a pandas data frame is reasonably simple:

[source,python]
----

import pandas as pd
df = pd.read_csv('foo.csv', index_col=False, header=0)
>>> df
   Foo
0    1
1    2
2    3
3    4
4    5
5    6
6    7
7    8
8    9
9   10
----

If we want to find the rows which have a value of 1 we'd write the following:

[source,python]
----

>>> df[df["Foo"] == 1]
   Foo
0    1
----

Finding the rows with a value less than 7 is as you'd expect too:

[source,python]
----

>>> df[df["Foo"] < 7]
   Foo
0    1
1    2
2    3
3    4
4    5
5    6
----

Next I wanted to filter out the rows containing odd numbers which I initially tried to do like this:

[source,python]
----

odds = [i for i in range(1,10) if i % 2 <> 0]
>>> odds
[1, 3, 5, 7, 9]

>>> df[df["Foo"] in odds]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/markneedham/projects/neo4j-himym/himym/lib/python2.7/site-packages/pandas/core/generic.py", line 698, in __nonzero__
    .format(self.__class__.__name__))
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
----

Unfortunately that doesn't work and I couldn't get any of the suggestions from the error message to work either. Luckily pandas has a special +++<cite>+++http://pandas.pydata.org/pandas-docs/dev/generated/pandas.DataFrame.isin.html[isin]+++</cite>+++ function for this use case which we can call like this:

[source,python]
----

>>> df[df["Foo"].isin(odds)]
   Foo
0    1
2    3
4    5
6    7
8    9
----

Much better!
R: Conditionally updating rows of a data frame
Date: 2015-02-26 00:45:42

In a blog post I wrote a couple of days ago about http://www.markhneedham.com/blog/2015/02/24/r-cohort-analysis-of-neo4j-meetup-members/[cohort analysis] I had to assign a monthNumber to each row in a data frame and started out with the following code:

[source,r]
----

library(zoo)
library(dplyr)

monthNumber = function(cohort, date) {
  cohortAsDate = as.yearmon(cohort)
  dateAsDate = as.yearmon(date)

  if(cohortAsDate > dateAsDate) {
    "NA"
  } else {
    paste(round((dateAsDate - cohortAsDate) * 12), sep="")
  }
}

cohortAttendance %>%
  group_by(row_number()) %>%
  mutate(monthNumber = monthNumber(cohort, date)) %>%
  filter(monthNumber != "NA") %>%
  filter(monthNumber != "0") %>%
  mutate(monthNumber = as.numeric(monthNumber)) %>%
  arrange(monthNumber)
----

If we time this function using +++<cite>+++system.time+++</cite>+++ we'll see that it's not very snappy:

[source,r]
----

system.time(cohortAttendance %>%
  group_by(row_number()) %>%
  mutate(monthNumber = monthNumber(cohort, date)) %>%
  filter(monthNumber != "NA") %>%
  filter(monthNumber != "0") %>%
  mutate(monthNumber = as.numeric(monthNumber)) %>%
  arrange(monthNumber))

   user  system elapsed
  1.968   0.019   2.016
----

The reason for the poor performance is that we process each row of the data table individually due to the call to +++<cite>+++group_by+++</cite>+++ on the second line. One way we can refactor the code is to use the +++<cite>+++ifelse+++</cite>+++ which can process multiple rows at a time:

[source,r]
----

system.time(
cohortAttendance %>%
  mutate(monthNumber = ifelse(as.yearmon(cohort) > as.yearmon(date),
                              paste((round(as.yearmon(date) - as.yearmon(cohort))*12), sep=""),
                              NA)))
   user  system elapsed
  0.026   0.000   0.026
----

https://twitter.com/tonkouts[Antonios] suggested another approach which involves first setting every row to 'NA' and then http://stackoverflow.com/questions/8214303/conditional-replacement-of-values-in-a-data-frame[selectively updating the appropriate rows]. I ended up with the following code:

[source,r]
----

cohortAttendance$monthNumber = NA

cohortAttendance$monthNumber[as.yearmon(cohortAttendance$cohort) > as.yearmon(cohortAttendance$date)] = paste((round(as.yearmon(cohortAttendance$date) - as.yearmon(cohortAttendance$cohort))*12), sep="")
----

Let's measure that:

[source,r]
----

system.time(paste((round(as.yearmon(cohortAttendance$date) - as.yearmon(cohortAttendance$cohort))*12), sep=""))
   user  system elapsed
  0.013   0.000   0.013
----

Both approaches are much quicker than my original version although this one seems to be marginally quicker than the ifelse approach.

Note to future Mark: try to avoid grouping by row number - there's usually a better and faster solution!
R/dplyr: Extracting data frame column value for filtering with %in%
Date: 2015-02-22 08:58:57

I've been playing around with http://cran.rstudio.com/web/packages/dplyr/vignettes/introduction.html[dplyr] over the weekend and wanted to extract the values from a data frame column to use in a later filtering step.

I had a data frame:

[source,r]
----

library(dplyr)
df = data.frame(userId = c(1,2,3,4,5), score = c(2,3,4,5,5))
----

And wanted to extract the userIds of those people who have a score greater than 3. I started with:

[source,r]
----

highScoringPeople = df %>% filter(score > 3) %>% select(userId)
> highScoringPeople
  userId
1      3
2      4
3      5
----

And then filtered the data frame expecting to get back those 3 people:

[source,r]
----

> df %>% filter(userId %in% highScoringPeople)
[1] userId score
<0 rows> (or 0-length row.names)
----

No rows! I created vector with the numbers 3-5 to make sure that worked:

[source,r]
----

> df %>% filter(userId %in% c(3,4,5))
  userId score
1      3     4
2      4     5
3      5     5
----

That works as expected so +++<cite>+++highScoringPeople+++</cite>+++ obviously isn't in the right format to facilitate an 'in lookup'. Let's explore:

[source,r]
----

> str(c(3,4,5))
 num [1:3] 3 4 5

> str(highScoringPeople)
'data.frame':	3 obs. of  1 variable:
 $ userId: num  3 4 5
----

Now it's even more obvious why it doesn't work - +++<cite>+++highScoringPeople+++</cite>+++ is still a data frame when we need it to be a vector/list.

One way to fix this is to extract the userIds using the $ syntax instead of the select function:

[source,r]
----

highScoringPeople = (df %>% filter(score > 3))$userId

> str(highScoringPeople)
 num [1:3] 3 4 5

> df %>% filter(userId %in% highScoringPeople)
  userId score
1      3     4
2      4     5
3      5     5
----

Or if we want to do the column selection using dplyr we can http://stackoverflow.com/questions/21618423/extract-a-dplyr-tbl-column-as-a-vector[extract the values for the column like this]:

[source,r]
----

highScoringPeople = (df %>% filter(score > 3) %>% select(userId))[[1]]

> str(highScoringPeople)
 num [1:3] 3 4 5
----

Not so difficult after all.