markhneedham/blog huggingface dataset:
Unix: tar - Extracting, creating and viewing archives
Date: 2013-08-22 22:56:23

I've been playing around with the Unix http://www.computerhope.com/unix/utar.htm[tar] command a bit this week and realised that I'd memorised some of the flag combinations but didn't actually know what each of them meant.

For example, one of the most common things that I want to do is extract a gripped neo4j archive:

[source,bash]
----

$ wget http://dist.neo4j.org/neo4j-community-1.9.2-unix.tar.gz
$ tar -xvf neo4j-community-1.9.2-unix.tar.gz
----

where:

* -x means extract
* -v means produce verbose output i.e. print out the names of all the files as you unpack it
* -f means unpack the file which follows this flag i.e. neo4j-community-1.9.2-unix.tar.gz in this example

I didn't realise that by default tar runs against standard input so we could actually achieve the above in one go with the following combination:</li> ~~~bash $ wget http://dist.neo4j.org/neo4j-community-1.9.2-unix.tar.gz -o - | tar -xv ~~~

The other thing I wanted to do was create a gripped archive from the contents of a folder, something which I do much less frequently and am therefore much more rusty at! The following does the trick: ~~~bash $ tar -cvzpf neo4j-football.tar.gz neo4j-football/ $ ls -alh neo4j-football.tar.gz -rw-r--r-- 1 markhneedham staff 526M 22 Aug 23:38 neo4j-football.tar.gz ~~~

where:

* -c means create a new archive
* -z means gzip that archive
* -p means preserve file permissions

Sometimes we'll want to exclude some things from our archive which is where the '--exclude' flag comes in handy.

For example I want to exclude the data, git and neo4j folders which sit inside 'neo4j-football' which I can do with the following: ~~~bash $ tar --exclude "data*" --exclude "neo4j-community*" --exclude ".git" -cvzpf neo4j-football.tar.gz neo4j-football/ $ ls -alh neo4j-football.tar.gz -rw-r--r-- 1 markhneedham staff 138M 22 Aug 23:36 neo4j-football.tar.gz ~~~

If we want to quickly check that our file has been created correctly we can run the following: ~~~bash $ tar -tvf neo4j-football.tar.gz ~~~

where:

* -t means list the contents of the archive to standard out

And that pretty much covers my main use cases for the moment!
Model to answer your questions rather than modelling reality
Date: 2013-08-22 21:26:10

On the recommendation of https://twitter.com/iansrobinson[Ian Robinson] I've been reading the 2nd edition of William's Kent's 'http://www.waterstones.com/waterstonesweb/products/william+kent/data+and+reality/5270709/[Data and Reality]' and the author makes an interesting observation at the end of the first chapter which resonated with me:

____
Once more: we are not modelling reality, but the way information about reality is processed, by people.
____

It reminds me of similar advice in Eric Evans' http://www.amazon.co.uk/Domain-driven-Design-Tackling-Complexity-Software/dp/0321125215[Domain Driven Design] and it's advice which I believe is helpful when designing a model in a graph database.

Last year I wrote a post explaining how I'd be using an approach of http://www.markhneedham.com/blog/2012/05/05/neo4j-what-question-do-you-want-to-answer/[defining questions that I wanted to ask] before modelling my data and in http://www.neo4j.org/[neo4j] land we can do this by writing http://docs.neo4j.org/chunked/snapshot/cypher-query-lang.html[cypher] queries up front.

We can then play around with increasing the size of our data set in different ways to check that our queries are still performant and tweak our model if necessary.

For example one simple optimisation would be to run an offline query to http://www.markhneedham.com/blog/2012/07/21/neo4j-embracing-the-sub-graph/[make implicit relationships explicit].

Although graphs are very http://neo4j.rubyforge.org/guides/why_graph_db.html[whiteboard friendly] and it can be tempting to design our whole model before writing any queries this often causes problems later on.

When we eventually get to asking questions of our data we may find that we've modelled some things unnecessarily or have designed the model in a way that leads to inefficient queries.

I've found an effective approach is to *keep the feedback loop tight* by minimising the amount of time between drawing parts of our model on a whiteboard and writing queries against it.

If you're interested in learning more, Ian has http://flex.winfxpro.info/download/?noderef=workspace://SpacesStore/c1d32b3e-67f1-4f0d-b5b9-b055ebdf9444[a slide deck from a talk he did at JAX 2013] which covers this idea and others when building out graph database applications.
Haskell: Reading files
Date: 2013-01-02 00:16:50

In writing the https://github.com/mneedham/algorithms2/blob/master/clustering.hs[clustering algorithm] which I've mentioned way too many times already I needed to process a text file which contained all the points and my initial approach looked like this:

[source,haskell]
----

import System.IO

main = do
    withFile "clustering2.txt" ReadMode (\handle -> do
        contents <- hGetContents handle
        putStrLn contents)
----

It felt a bit clunky but I didn't realise there was an easier way until I came across http://stackoverflow.com/questions/7867723/haskell-file-reading[this thread]. We can simplify reading a file to the following by using the +++<cite>+++http://zvon.org/other/haskell/Outputprelude/readFile_f.html[readFile]+++</cite>+++ function:

[source,haskell]
----

main = do
    contents <- readFile "clustering2.txt"
    putStrLn contents
----

We need to read the file in the IO monad which explains why we have the 'do' notation on the first line.

Another thing I http://www.haskell.org/ghc/docs/7.2.2/html/users_guide/interactive-evaluation.html[didn't realise until recently] was that you don't actually need to worry about the 'do' notation if you try to read from the IO monad inside GHCI.

In this context we're reading from the IO monad when we bind 'readFile' to the variable 'contents' since 'readFile' returns type 'IO String':

[source,text]
----

> :t readFile
readFile :: FilePath -> IO String
----

We can therefore play around with the code pretty easily:

[source,haskell]
----

> contents <- readFile "clustering2.txt"
> let (bits, nodes) = process contents
> bits
24
> length nodes
19981
> take 10 nodes
[379,1669,5749,6927,7420,9030,9188,9667,11878,12169]
----

I think we're able to do this because by being in GHCI we're already in the context of the IO monad but I'm happy to be corrected if I haven't explained that correctly.
telnet/netcat: Waiting for a port to be open
Date: 2013-01-20 15:53:02

On Friday http://junctionbox.ca/[Nathan] and I were setting up a new virtual machine and we needed a firewall rule to be created to allow us to connect to another machine which had some JAR files we wanted to download.

We wanted to know when it had been done by one of our operations team and I initially thought we might be able to do that using telnet:

[source,text]
----

$ telnet 10.0.0.1 8081
Trying 10.0.0.1...
telnet: connect to address 10.0.0.1: Operation timed out
telnet: Unable to connect to remote host
----

We wanted to put a http://en.wikipedia.org/wiki/Watch_(Unix)[watch] on the command so that it would be repeated every few seconds and indicate when we'd could connect to the port. However, as as far as I can tell there's no way to reduce the length of the telnet timeout so Nathan suggested using netcat instead.

We ended up with the following command...

[source,text]
----

$ nc -v -w 1 10.0.0.1 8081
nc: connect to 10.0.0.1 port 8081 (tcp) failed: Connection refused
----

\...which we can then wire up with watch like so:

[source,text]
----

$ watch "nc -v -w 1 10.0.0.1 8081"

Every 2.0s: nc -v -w 1 10.0.0.1 8081                         Sun Jan 20 15:48:05 2013

nc: connect to 10.0.0.1 port 8081 (tcp) timed out: Operation now in progress
----

And then when it works:

[source,text]
----

Every 2.0s: nc -v -w 1 10.0.0.1 8081                         Sun Jan 20 15:49:53 2013

Connection to 10.0.0.1 8081 port [tcp] succeeded!
----
Ruby: invalid multibyte char (US-ASCII)
Date: 2013-01-27 15:14:01

I've used Ruby on and off for the last few years but somehow had never come across the following error which we got last week while attempting to print out a currency value:

+++<cite>+++blah.ruby+++</cite>+++

[source,ruby]
----

amount = "£10.00"
puts amount
----

[source,text]
----

$ ruby blah.ruby
blah.ruby:1: invalid multibyte char (US-ASCII)
blah.ruby:1: invalid multibyte char (US-ASCII)
----

Luckily my pair http://www.linkedin.com/profile/view?id=13286237[Jae] had come across this before and showed me http://blog.grayproductions.net/articles/ruby_19s_three_default_encodings[a blog post which explains what's going on and how to sort it out].

It turns out that since Ruby 1.9 we need to explicitly specify which encoding we want to use for strings otherwise it will default to US-ASCII in my case at least.

We can get around this by explicitly telling the interpreter that we want to use UTF-8 encoding by including the following comment at the top of our file:

[source,ruby]
----

# encoding: UTF-8
amount = "£10.00"
puts amount
----

Then when we interpret the file this time:

[source,text]
----

$ ruby blah.ruby
£10.00
----

Since we were only editing one file it made sense to just set the comment manually but if we need to do that in a more widespread way then we'd want to use the https://github.com/m-ryan/magic_encoding[magic_encoding gem].

Yehuda Katz also wrote a post a few years ago http://yehudakatz.com/2010/05/05/ruby-1-9-encodings-a-primer-and-the-solution-for-rails/[explaining all things encoding in more detail].
Python/numpy: Selecting values by multiple indices
Date: 2013-01-27 02:21:39

As I http://www.markhneedham.com/blog/2013/01/27/pythonnumpy-selecting-specific-column-in-2d-array/[mentioned in my previous post] I've been playing around with http://www.numpy.org/[numpy] and I wanted to get the values of a collection of different indices in a 2D array.

If we had a 2D array that looked like this:

[source,python]
----

>>> x = arange(20).reshape(4,5)
>>> x
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
----

I knew that it was possible to retrieve the first 3 rows by using the following code:

[source,python]
----

>>> x[0:3]
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])
----

What I wanted to do, however, was retrieve the 1st, 3rd and 4th rows which we can do by passing a collection to the array lookup function:

[source,python]
----

>>> x[[0,2,3]]
array([[ 0,  1,  2,  3,  4],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
----

My collection of indices was actually in a tuple so I needed to use the +++<cite>+++http://docs.python.org/2/library/functions.html#list[list]+++</cite>+++ function to convert it to the appropriate data structure first:

[source,python]
----

>>> x[list((0,2,3))]
array([[ 0,  1,  2,  3,  4],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
----

Pretty neat!
Python: (Conceptually) removing an item from a tuple
Date: 2013-01-27 02:30:05

As part of some code I've been playing around I wanted to remove an item from a tuple which wasn't particularly easy because Python's tuple data structure is immutable.

I therefore needed to create a new tuple excluding the value which I wanted to remove.

I ended up writing the following function to do this but I imagine there might be an easier way because it's quite verbose:

[source,python]
----

def tuple_without(original_tuple, element_to_remove):
    new_tuple = []
    for s in list(original_tuple):
        if not s == element_to_remove:
            new_tuple.append(s)
    return tuple(new_tuple)
----

Which can be used like so:

[source,python]
----

>>> tuple_without((1,2,3,4), 1)
(2, 3, 4)
----

[source,python]
----

>>> tuple_without((1,2,3,4), 0)
(1, 2, 3, 4)
----

The easiest approach seemed to be to build up a list containing all the values and then convert it to a tuple by using the +++<cite>+++http://docs.python.org/2/library/functions.html#tuple[tuple]+++</cite>+++ function.

It'd be cool if there was a way to transform a tuple like this but I couldn't find such a function in my travels.
Python/numpy: Selecting specific column in 2D array
Date: 2013-01-27 02:10:10

I've been playing around with http://www.numpy.org/[numpy] this evening in an attempt to improve the performance of a http://en.wikipedia.org/wiki/Travelling_salesman_problem[Travelling Salesman Problem implementation] and I wanted to get every value in a specific column of a 2D array.

The array looked something like this:

[source,python]
----

>>> x = arange(20).reshape(4,5)
>>> x
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])
----

I wanted to get the values for the 2nd column of each row which would return an array containing 1, 6, 11 and 16.

For some reason I was expecting it to be quite complicated but in fact we can do this by http://stackoverflow.com/questions/2111163/selecting-specific-column-in-each-row-from-array[using matrix style syntax] like so:

[source,python]
----

>>> x[:, 1]
array([ 1,  6, 11, 16])
----

Here we are first saying that we want to return all the rows by specifying ':' and then the '1' indicates that we only want to return the column with index 1.

If we wanted to return a specific row as well then we'd specify a value before the comma and it'd be a standard 2D array value lookup:

[source,python]
----

>> x[2,1]
11
----

or

[source,python]
----

>> x[2][1]
11
----
Kaggle Digit Recognizer: A feature extraction #fail
Date: 2013-01-31 23:24:55

I've written http://www.markhneedham.com/blog/tag/kaggle/[a few blog posts] about our attempts at the http://www.kaggle.com/c/digit-recognizer[Kaggle Digit Recogniser] problem and one thing we haven't yet tried is feature extraction.

Feature extraction in this context means that we'd generate some other features to train a classifier with rather than relying on just the pixel values we were provided.

Every week https://twitter.com/jennifersmithco[Jen] would try and persuade me that we should try it out but it wasn't until I was flicking through http://columbiadatascience.com/2012/10/15/10-important-data-science-ideas/[the notes from the Columbia Data Science class] that it struck home:

____
The Space between the Data Set and the Algorithm *Many people go straight from a data set to applying an algorithm*. But there's a huge space in between of important stuff. It's easy to run a piece of code that predicts or classifies. That's not the hard part. The hard part is doing it well. One needs to conduct exploratory data analysis as I've emphasized; and conduct feature selection as Will Cukierski emphasized.
____

I've highlighted the part of the post which describes exactly what we've been doing!

There were some http://www.kaggle.com/c/digit-recognizer/forums/t/2308/feature-extraction-technique[examples of feature extraction] on the Kaggle forums so I thought I'd try and create some other features using http://www.markhneedham.com/blog/category/r/[R].

I created features for the number of non zero pixels, the number of 255 pixels, the average number of pixels and the average of the middle pixels of a number.

The code reads like this:

[source,r]
----

initial <- read.csv("train.csv", header = TRUE)
initial$nonZeros <- apply(initial, 1, function(entries) length(Filter(function (x) x != 0, entries)))
initial$fullHouses <- apply(initial, 1, function(entries) length(Filter(function (x) x == 255, entries)))
initial$meanPixels <- apply(initial, 1, mean)
initial$middlePixels <- apply(initial[,200:500], 1, mean)
----

I then wrote those features out into a CSV file like so:

[source,r]
----

newFeatures <- subset(initial, select=c(label, nonZeros, meanPixels, fullHouses, middlePixels))
write.table(file="feature-extraction.txt", newFeatures, row.names=FALSE, sep=",")
----

I then created a http://www.markhneedham.com/blog/2012/12/27/mahout-parallelising-the-creation-of-decisiontrees/[100 tree random forest using Mahout] to see whether or not we could get any sort of accuracy using these features.

Unfortunately the accuracy on the cross validation set (10% of the training data) was only 24% which is pretty useless so it's back to the drawing board!

Our next task is to try and work out whether we can derive some features which have a stronger correlation with the label values or combining the new features with the existing pixel values to see if that has any impact.

As you can probably tell I don't really understand how you should go about extracting features so if anybody has ideas or papers/articles I can read to learn more please let me know in the comments!
R: Ordering rows in a data frame by multiple columns
Date: 2013-01-23 23:09:28

In one of the assignments of https://www.coursera.org/course/compdata[Computing for Data Analysis] we needed to sort a data frame based on the values in two of the columns and then return the top value.

The initial data frame looked a bit like this:

[source,r]
----

> names <- c("paul", "mark", "dave", "will", "john")
> values <- c(1,4,1,2,1)
> smallData <- data.frame(name = names, value = values)
> smallData
  name value
1 paul     1
2 mark     4
3 dave     1
4 will     2
5 john     1
----

I want to be able to http://stackoverflow.com/questions/1296646/how-to-sort-a-dataframe-by-columns-in-r[sort the data frame by value and name both in ascending order] so the final result should look like this:

[source,text]
----

  name value
3 dave     1
5 john     1
1 paul     1
4 will     2
2 mark     4
----

To do that we can use the +++<cite>+++http://stat.ethz.ch/R-manual/R-patched/library/base/html/order.html[order]+++</cite>+++ function which will tell us the indices of the vector in sorted order.

e.g. in our case

[source,r]
----

> order(c(1,4,1,2,1))
[1] 1 3 5 4 2
----

If we pass a collection of indices to the extract operation it'll reorder the rows. e.g.

[source,r]
----

> smallData[c(5,4,3,2,1),]
  name value
5 john     1
4 will     2
3 dave     1
2 mark     4
1 paul     1
----

In our case we wire everything together like this to sort by the second column (value):

[source,r]
----

> smallData[order(smallData[,2]),]
  name value
1 paul     1
3 dave     1
5 john     1
4 will     2
2 mark     4
----

It's a reasonably small tweak to get it to sort first by the second column and then by the first (name) which is what we want:

[source,r]
----

> smallData[order(smallData[,2], smallData[,1]),]
  name value
3 dave     1
5 john     1
1 paul     1
4 will     2
2 mark     4
----

If we wanted to use the column names instead of indices we'd do the following:

[source,r]
----

> smallData[order(smallData$value, smallData$name),]
  name value
3 dave     1
5 john     1
1 paul     1
4 will     2
2 mark     4
----

We could also rewrite it using the +++<cite>+++with+++</cite>+++ function if we want to reduce the code further:

[source,r]
----

> smallData[with(smallData, order(value, name)),]
  name value
3 dave     1
5 john     1
1 paul     1
4 will     2
2 mark     4
----

As I understand it, when we use +++<cite>+++with+++</cite>+++ we put +++<cite>+++smallData+++</cite>+++ into the environment and evaluate the second argument to +++<cite>+++with+++</cite>+++ with respect to that so in this case it allows us to refer to the column names of +++<cite>+++smallData+++</cite>+++.
R: Filter a data frame based on values in two columns
Date: 2013-01-23 22:34:01

In the most recent assignment of the https://www.coursera.org/course/compdata[Computing for Data Analysis] course we had to filter a data frame which contained N/A values in two columns to only return rows which had no N/A's.

I started with a data frame that looked like this:

[source,r]
----

> data <- read.csv("specdata/002.csv")
> # we'll just use a few rows to make it easier to see what's going on
> data[2494:2500,]
           Date sulfate nitrate ID
2494 2007-10-30    3.25   0.902  2
2495 2007-10-31      NA      NA  2
2496 2007-11-01      NA      NA  2
2497 2007-11-02    6.56   1.270  2
2498 2007-11-03      NA      NA  2
2499 2007-11-04      NA      NA  2
2500 2007-11-05    6.10   0.772  2
----

We want to only return the rows which have a value in both the 'sulfate' and the 'nitrate' columns.

I initially tried to use the +++<cite>+++http://www.r-bloggers.com/filtering-a-list-with-the-filter-higher-order-function/[Filter]+++</cite>+++ function but wasn't very successful:

[source,r]
----

> smallData <- data[2494:2500,]
> Filter(function(x) !is.na(x$sulfate), smallData)
Error in x$sulfate : $ operator is invalid for atomic vectors
----

I'm not sure that +++<cite>+++Filter+++</cite>+++ is designed to filter data frames - it seems more appropriate for lists or vectors - so I ended up filtering the data frame using what I think is called an extract operation:

[source,r]
----

> smallData[!is.na(smallData$sulfate) & !is.na(smallData$nitrate),]
           Date sulfate nitrate ID
2494 2007-10-30    3.25   0.902  2
2497 2007-11-02    6.56   1.270  2
2500 2007-11-05    6.10   0.772  2
----

The code inside the square brackets returns a collection indicating whether or not we should return each row:

[source,r]
----

> !is.na(smallData$sulfate) & !is.na(smallData$nitrate)
[1]  TRUE FALSE FALSE  TRUE FALSE FALSE  TRUE
----

which is equivalent to doing this:

[source,r]
----

> smallData[c(TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, TRUE),]
           Date sulfate nitrate ID
2494 2007-10-30    3.25   0.902  2
2497 2007-11-02    6.56   1.270  2
2500 2007-11-05    6.10   0.772  2
----

We put a comma after the list of true/false values to indicate that we want to return all the columns otherwise we'd get this error:

[source,r]
----

> smallData[c(TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, TRUE)]
Error in `[.data.frame`(smallData, c(TRUE, FALSE, FALSE, TRUE, FALSE,  :
  undefined columns selected
----

We could filter the columns as well if we wanted to:

[source,r]
----

> smallData[c(TRUE, FALSE, FALSE, TRUE, FALSE, FALSE, TRUE), c(1,2)]
           Date sulfate
2494 2007-10-30    3.25
2497 2007-11-02    6.56
2500 2007-11-05    6.10
----

As is no doubt obvious, I don't know much R so if there's a better way to do anything please let me know.

The https://github.com/mneedham/dataanalysis/blob/master/week2/complete.R[full code is on github] if you're interested in seeing it in context.