markhneedham/blog huggingface dataset:
Python: scikit-learn: ImportError: cannot import name __check_build
Date: 2015-01-10 08:48:04

In https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-3-more-fun-with-word-vectors[part 3 of Kaggle's series on text analytics] I needed to install http://scikit-learn.org/stable/[scikit-learn] and having done so ran into the following error when trying to use one of its classes:

[source,python]
----

>>> from sklearn.feature_extraction.text import CountVectorizer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/markneedham/projects/neo4j-himym/himym/lib/python2.7/site-packages/sklearn/__init__.py", line 37, in <module>
    from . import __check_build
ImportError: cannot import name __check_build
----

This error doesn't reveal very much but I found that when I exited the REPL and tried the same command again I got a different error which was a bit more useful:

[source,python]
----

>>> from sklearn.feature_extraction.text import CountVectorizer
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/markneedham/projects/neo4j-himym/himym/lib/python2.7/site-packages/sklearn/__init__.py", line 38, in <module>
    from .base import clone
  File "/Users/markneedham/projects/neo4j-himym/himym/lib/python2.7/site-packages/sklearn/base.py", line 10, in <module>
    from scipy import sparse
ImportError: No module named scipy
----

The fix for this is now obvious:

[source,python]
----

$ pip install scipy
----

And I can now load +++<cite>+++CountVectorizer+++</cite>+++ without any problem:

[source,python]
----

$ python
Python 2.7.5 (default, Aug 25 2013, 00:04:04)
[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from sklearn.feature_extraction.text import CountVectorizer
----
R: ggplot2 - Each group consist of only one observation. Do you need to adjust the group aesthetic?
Date: 2015-01-30 00:27:53

I've been playing around with some weather data over the last couple of days which I aggregated down to the average temperature per month over the last 4 years and stored in a CSV file.

This is what the file looks like:

[source,bash]
----

$ cat /tmp/averageTemperatureByMonth.csv
"month","aveTemperature"
"January",6.02684563758389
"February",5.89380530973451
"March",7.54838709677419
"April",10.875
"May",13.3064516129032
"June",15.9666666666667
"July",18.8387096774194
"August",18.3709677419355
"September",16.2583333333333
"October",13.4596774193548
"November",9.19166666666667
"December",7.01612903225806
----

I wanted to create a simple line chart which would show the months of the year in ascending order with the appropriate temperature.

My first attempt was the following:

[source,rstats]
----

df = read.csv("/tmp/averageTemperatureByMonth.csv")
df$month = factor(df$month, month.name)

ggplot(aes(x = month, y = aveTemperature), data = df) +
  geom_line( ) +
  ggtitle("Temperature by month")
----

which resulted in the following error:

[source,text]
----

geom_path: Each group consist of only one observation. Do you need to adjust the group aesthetic?
----

My understanding is that the points don't get joined up by default because the variable on the x axis is not a continuous one but rather a factor variable.

One way to work around this problem is to make it numeric, like so:

[source,rstats]
----

ggplot(aes(x = as.numeric(month), y = aveTemperature), data = df) +
  geom_line( ) +
  ggtitle("Temperature by month")
----

which results in the following chart:

image::{{<siteurl>}}/uploads/2015/01/2015-01-30_00-25-18.png[2015 01 30 00 25 18,600]

This isn't bad but it'd be much nicer if we could have the month names along the bottom instead.

It turns out we can but we need to https://kohske.wordpress.com/2010/12/27/faq-geom_line-doesnt-draw-lines/[specify a group] that each point belongs to. ggplot will then connects points which belong to the same group.

In this case we don't really have one so we'll define a dummy one instead:

[source,rstats]
----

ggplot(aes(x = month, y = aveTemperature, group=1), data = df) +
  geom_line( ) +
  ggtitle("Temperature by month")
----

And now we get the visualisation we want:

image::{{<siteurl>}}/uploads/2015/01/2015-01-29_23-28-23.png[2015 01 29 23 28 23,600]
Python: Counter - ValueError: too many values to unpack
Date: 2015-01-12 23:16:58

I recently came across https://docs.python.org/2/library/collections.html#collections.Counter[Python's Counter tool] which makes it really easy to count the number of occurrences of items in a list.

In my case I was trying to work out how many times words occurred in a corpus so I had something like the following:

[source,python]
----

>> from collections import Counter
>> counter = Counter(["word1", "word2", "word3", "word1"])
>> print counter
Counter({'word1': 2, 'word3': 1, 'word2': 1})
----

I wanted to write a for loop to iterate over the counter and print the (key, value) pairs and started with the following:

[source,python]
----

>>> for key, value in counter:
...   print key, value
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: too many values to unpack
----

I'm not sure why I expected this to work but in fact since Counter is a sub class of dict we need to call https://docs.python.org/2/library/stdtypes.html#dict.iteritems[iteritems] to get an iterator of pairs rather than just keys.

The following does the job:

[source,python]
----

>>> for key, value in counter.iteritems():
...   print key, value
...
word1 2
word3 1
word2 1
----

Hopefully future Mark will remember this!
Netty: Testing encoders/decoders
Date: 2015-06-05 21:25:25

I've been working with Netty a bit recently and having built a pipeline of encoders/decoders as described in this http://seeallhearall.blogspot.co.uk/2012/05/netty-tutorial-part-1-introduction-to.html[excellent tutorial] wanted to test that the encoders and decoders were working without having to send real messages around.

Luckily there is a https://netty.io/4.0/api/io/netty/channel/embedded/EmbeddedChannel.html[EmbeddedChannel] which makes our life very easy indeed.

Let's say we've got a message 'Foo' that we want to send across the wire. It only contains a single integer value so we'll just send that and reconstruct 'Foo' on the other side.

We might write the following encoder to do this:

[source,java]
----

// Examples uses Netty 4.0.28.Final
public static class MessageEncoder extends MessageToMessageEncoder<Foo>
{
    @Override
    protected void encode( ChannelHandlerContext ctx, Foo msg, List<Object> out ) throws Exception
    {
        ByteBuf buf = ctx.alloc().buffer();
        buf.writeInt( msg.value() );
        out.add( buf );
    }
}

public static class Foo
{
    private Integer value;

    public Foo(Integer value)
    {
        this.value = value;
    }

    public int value()
    {
        return value;
    }
}
----

So all we're doing is taking the 'value' field out of 'Foo' and putting it into the list which gets passed downstream.

Let's write a test which simulates sending a 'Foo' message and has an empty decoder attempt to process the message:

[source,java]
----

@Test
public void shouldEncodeAndDecodeVoteRequest()
{
    // given
    EmbeddedChannel channel = new EmbeddedChannel( new MessageEncoder(), new MessageDecoder() );

    // when
    Foo foo = new Foo( 42 );
    channel.writeOutbound( foo );
    channel.writeInbound( channel.readOutbound() );

    // then
    Foo returnedFoo = (Foo) channel.readInbound();
    assertNotNull(returnedFoo);
    assertEquals( foo.value(), returnedFoo.value() );
}

public static class MessageDecoder extends MessageToMessageDecoder<ByteBuf>
{
    @Override
    protected void decode( ChannelHandlerContext ctx, ByteBuf msg, List<Object> out ) throws Exception { }
}
----

So in the test we write 'Foo' to the outbound channel and then read it back into the inbound channel and then check what we've got. If we run that test now this is what we'll see:

[source,java]
----

junit.framework.AssertionFailedError
	at NettyTest.shouldEncodeAndDecodeVoteRequest(NettyTest.java:28)
----

The message we get back is null which makes sense given that we didn't bother writing the decoder. Let's implement the decoder then:

[source,java]
----

public static class MessageDecoder extends MessageToMessageDecoder<ByteBuf>
{
    @Override
    protected void decode( ChannelHandlerContext ctx, ByteBuf msg, List<Object> out ) throws Exception
    {
        int value = msg.readInt();
        out.add( new Foo(value) );
    }
}
----

Now if we run our test again it's all green and happy. We can now go and encode/decode some more complex structures and update our test accordingly.
R: dplyr - removing empty rows
Date: 2015-06-02 06:49:10

I'm still working my way through the exercises in Think Bayes and in Chapter 6 needed to do some cleaning of the data in a CSV file containing information about the Price is Right.

I downloaded the file using wget:

[source,bash]
----

wget ￼http://www.greenteapress.com/thinkbayes/showcases.2011.csv￼
----

And then loaded it into R and explored the first few rows using dplyr

[source,R]
----

library(dplyr)
df2011 = read.csv("~/projects/rLearning/showcases.2011.csv")

> df2011 %>% head(10)

           X Sep..19 Sep..20 Sep..21 Sep..22 Sep..23 Sep..26 Sep..27 Sep..28 Sep..29 Sep..30 Oct..3
1              5631K   5632K   5633K   5634K   5635K   5641K   5642K   5643K   5644K   5645K  5681K
2
3 Showcase 1   50969   21901   32815   44432   24273   30554   20963   28941   25851   28800  37703
4 Showcase 2   45429   34061   53186   31428   22320   24337   41373   45437   41125   36319  38752
5
...
----

As you can see, we have some empty rows which we want to get rid of to ease future processing. I couldn't find an easy way to filter those out but what we can do instead is have empty columns converted to 'NA' and then filter those.

First we need to tell +++<cite>+++read.csv+++</cite>+++ to treat empty columns as NA:

[source,r]
----

df2011 = read.csv("~/projects/rLearning/showcases.2011.csv", na.strings = c("", "NA"))
----

And now we can filter them out using +++<cite>+++na.omit+++</cite>+++:

[source,r]
----

df2011 = df2011 %>% na.omit()

> df2011  %>% head(5)
             X Sep..19 Sep..20 Sep..21 Sep..22 Sep..23 Sep..26 Sep..27 Sep..28 Sep..29 Sep..30 Oct..3
3   Showcase 1   50969   21901   32815   44432   24273   30554   20963   28941   25851   28800  37703
4   Showcase 2   45429   34061   53186   31428   22320   24337   41373   45437   41125   36319  38752
6        Bid 1   42000   14000   32000   27000   18750   27222   25000   35000   22500   21300  21567
7        Bid 2   34000   59900   45000   38000   23000   18525   32000   45000   32000   27500  23800
9 Difference 1    8969    7901     815   17432    5523    3332   -4037   -6059    3351    7500  16136
...
----

Much better!
R: dplyr - segfault cause 'memory not mapped'
Date: 2015-06-20 22:18:55

In my http://www.markhneedham.com/blog/2015/06/19/r-regex-capturing-multiple-matches-of-the-same-group/[continued playing around with web logs in R] I wanted to process the logs for a day and see what the most popular URIs were.

I first read in all the lines using the +++<cite>+++read_lines+++</cite>+++ function in readr and put the vector it produced into a data frame so I could process it using dplyr.

[source,r]
----

library(readr)
dlines = data.frame(column = read_lines("~/projects/logs/2015-06-18-22-docs"))
----

In the previous post I showed some code to extract the URI from a log line. I extracted this code out into a function and adapted it so that I could pass in a list of values instead of a single value:

[source,r]
----

extract_uri = function(log) {
  parts = str_extract_all(log, "\"[^\"]*\"")
  return(lapply(parts, function(p) str_match(p[1], "GET (.*) HTTP")[2] %>% as.character))
}
----

Next I ran the following function to count the number of times each URI appeared in the logs:

[source,r]
----

library(dplyr)
pages_viewed = dlines %>%
  mutate(uri  = extract_uri(column)) %>%
  count(uri) %>%
  arrange(desc(n))
----

This crashed my R process with the following error message:

[source,r]
----

segfault cause 'memory not mapped'
----

I narrowed it down to a problem when doing a group by operation on the 'uri' field and came across https://github.com/hadley/dplyr/issues/322[this post] which suggested that it was handled more cleanly in more recently version of dplyr.

I upgraded to 0.4.2 and tried again:

[source,R]
----

## Error in eval(expr, envir, enclos): cannot group column uri, of class 'list'
----

That makes more sense. We're probably returning a list from +++<cite>+++extract_uri+++</cite>+++ rather than a vector which would fit nicely back into the data frame. That's fixed easily enough by unlisting the result:

[source,r]
----

extract_uri = function(log) {
  parts = str_extract_all(log, "\"[^\"]*\"")
  return(unlist(lapply(parts, function(p) str_match(p[1], "GET (.*) HTTP")[2] %>% as.character)))
}
----

And now when we run the count function it's happy again, good times!
R: dplyr - squashing multiple rows per group into one
Date: 2015-06-27 22:36:50

I spent a bit of the day working on my http://www.markhneedham.com/blog/2015/06/25/r-scraping-wimbledon-draw-data/[Wimbledon] http://www.markhneedham.com/blog/2015/06/26/r-ggplot-show-discrete-scale-even-with-no-value/[data set] and the next thing I explored is all the people that have beaten Andy Murray in the tournament.

The following dplyr query gives us the names of those people and the year the match took place:

[source,r]
----

library(dplyr)

> main_matches %>% filter(loser == "Andy Murray") %>% select(winner, year)

            winner year
1  Grigor Dimitrov 2014
2    Roger Federer 2012
3     Rafael Nadal 2011
4     Rafael Nadal 2010
5     Andy Roddick 2009
6     Rafael Nadal 2008
7 Marcos Baghdatis 2006
8 David Nalbandian 2005
----

As you can see, Rafael Nadal shows up multiple times. I wanted to get one row per player and list all the years in a single column.

This was my initial attempt:

[source,r]
----

> main_matches %>% filter(loser == "Andy Murray") %>%
     group_by(winner) %>% summarise(years = paste(year))
Source: local data frame [6 x 2]

            winner years
1     Andy Roddick  2009
2 David Nalbandian  2005
3  Grigor Dimitrov  2014
4 Marcos Baghdatis  2006
5     Rafael Nadal  2011
6    Roger Federer  2012
----

Unfortunately it just gives you the last matching row per group which isn't quite what we want.. I realised my mistake while trying to pass a vector into paste and noticing that a vector came back when I'd expected a string:

[source,r]
----

> paste(c(2008,2009,2010))
[1] "2008" "2009" "2010"
----

The missing argument was 'collapse' - http://www.markhneedham.com/blog/2014/08/11/r-grouping-by-two-variables/[something I'd come across when using plyr] last year:

[source,r]
----

> paste(c(2008,2009,2010), collapse=", ")
[1] "2008, 2009, 2010"
----

Now, if we apply that to our original function:

[source,r]
----

> main_matches %>% filter(loser == "Andy Murray") %>%
     group_by(winner) %>% summarise(years = paste(year, collapse=", "))
Source: local data frame [6 x 2]

            winner            years
1     Andy Roddick             2009
2 David Nalbandian             2005
3  Grigor Dimitrov             2014
4 Marcos Baghdatis             2006
5     Rafael Nadal 2011, 2010, 2008
6    Roger Federer             2012
----

That's exactly what we want. Let's tidy that up a bit:

[source,r]
----

> main_matches %>% filter(loser == "Andy Murray") %>%
     group_by(winner) %>% arrange(year) %>%
     summarise(years  = paste(year, collapse =","), times = length(year))  %>%
     arrange(desc(times), years)
Source: local data frame [6 x 3]

            winner          years times
1     Rafael Nadal 2008,2010,2011     3
2 David Nalbandian           2005     1
3 Marcos Baghdatis           2006     1
4     Andy Roddick           2009     1
5    Roger Federer           2012     1
6  Grigor Dimitrov           2014     1
----
R: Command line - Error in GenericTranslator$new : could not find function "loadMethod"
Date: 2015-06-27 22:47:22

I've been reading https://pragprog.com/book/rmtpruby/text-processing-with-ruby[Text Processing with Ruby] over the last week or so and one of the ideas the author describes is setting up your scripts so you can run them directly from the command line.

I wanted to do this with my Wimbledon R script and wrote the following script which uses the 'Rscript' executable so that R doesn't launch in interactive mode:

_wimbledon_

[source,bash]
----

#!/usr/bin/env Rscript

library(rvest)
library(dplyr)
library(stringr)
library(readr)

# stuff
----

Then I tried to run it:

[source,bash]
----

$ time ./wimbledon

...

Error in GenericTranslator$new : could not find function "loadMethod"
Calls: write.csv ... html_extract_n -> <Anonymous> -> Map -> mapply -> <Anonymous> -> $
Execution halted

real	0m1.431s
user	0m1.127s
sys	0m0.078s
----

As the error suggests, the script fails when trying to write to a CSV file - it looks like Rscript doesn't load in something from the core library that we need. It turns out https://groups.google.com/forum/#!topic/shiny-discuss/Gx2P_dhzM38[adding the following line] to our script is all we need:

[source,bash]
----

library(methods)
----

So we end up with this:

[source,bash]
----

#!/usr/bin/env Rscript

library(methods)
library(rvest)
library(dplyr)
library(stringr)
library(readr)
----

And when we run that all is well!
Mac OS X: GNU sed -  Hex string replacement / replacing new line characters
Date: 2015-06-11 21:38:32

Recently I was working with a CSV file which contained both Windows and Unix line endings which was making it difficult to work with.

The actual line endings were HEX '0A0D' i.e. Windows line breaks but there were also HEX 'OA' i.e. Unix line breaks within one of the columns.

I wanted to get rid of the Unix line breaks and discovered that you can do http://stackoverflow.com/questions/7760717/hex-string-replacement-using-sed[HEX sequence replacement using the GNU version of sed] - unfortunately the Mac ships with the BSD version which doesn't have this functionaltiy.

The first step was therefore to install the GNU version of sed.

[source,bash]
----

brew install coreutils
brew install gnu-sed --with-default-names
----

I wanted to replace my system sed so that's why I went with the '--with-default-names' flag - without that flag I believe the sed installation would be accessible as 'gs-sed'.

The following is an example of what the lines in the file look like:

[source,bash]
----

$ echo -e "Hello\x0AMark\x0A\x0D"
Hello
Mark
----

We want to get rid of the new line in between 'Hello' and 'Mark' but leave the other one be. I adapted one of the commands from http://backreference.org/2009/12/23/how-to-match-newlines-in-sed/[this tutorial] to look for lines which end in '0A' where that isn't followed by a '0D':

[source,bash]
----

$ echo -e "Hello\x0AMark\x0A\x0D" | \
  sed 'N;/\x0A[^\x0D]/s/\n/ /'
Hello Mark
----

Let's go through the parts of the sed command:

* +++<cite>+++N+++</cite>+++ - this creates a multiline pattern space by reading a new line of input and appending it to the contents of the pattern space. The two lines are separated by a new line.
* +++<cite>+++/\x0A[{caret}\x0D]/+++</cite>+++ - this matches any lines which contain 'OA' not followed by 'OD'
* +++<cite>+++/s/\n/ /+++</cite>+++ - this substitutes the new line character with a space for those matching lines from the previous command.

Now let's check it works if we have multiple lines that we want to squash:

[source,bash]
----

$ echo -e "Hello\x0AMark\x0A\x0DHello\x0AMichael\x0A\x0D"
Hello
Mark
Hello
Michael

$ echo -e "Hello\x0AMark\x0A\x0DHello\x0AMichael\x0A\x0D" | \
  sed 'N;/\x0A[^\x0D]/s/\n/ /'
Hello Mark
Hello Michael
----

Looks good! The actual file is a bit more nuanced so I've still got a bit more work to do but this is a good start.
R: dplyr - Update rows with earlier/previous rows values
Date: 2015-06-28 22:30:08

Recently I had a data frame which contained a column which had mostly empty values:

[source,r]
----

> data.frame(col1 = c(1,2,3,4,5), col2  = c("a", NA, NA , "b", NA))
  col1 col2
1    1    a
2    2 <NA>
3    3 <NA>
4    4    b
5    5 <NA>
----

I wanted to fill in the NA values with the last non NA value from that column. So I want the data frame to look like this:

[source,text]
----

1    1    a
2    2    a
3    3    a
4    4    b
5    5    b
----

I spent ages searching around before I http://stackoverflow.com/questions/27207162/fill-in-na-based-on-the-last-non-na-value-for-each-group-in-r[came across the +++<cite>+++na.locf+++</cite>+++ function in the zoo library] which does the job:

[source,r]
----

library(zoo)
library(dplyr)

> data.frame(col1 = c(1,2,3,4,5), col2  = c("a", NA, NA , "b", NA)) %>%
    do(na.locf(.))
  col1 col2
1    1    a
2    2    a
3    3    a
4    4    b
5    5    b
----

This will fill in the missing values for every column, so if we had a third column with missing values it would populate those too:

[source,r]
----

> data.frame(col1 = c(1,2,3,4,5), col2  = c("a", NA, NA , "b", NA), col3 = c("A", NA, "B", NA, NA)) %>%
    do(na.locf(.))

  col1 col2 col3
1    1    a    A
2    2    a    A
3    3    a    B
4    4    b    B
5    5    b    B
----

If we only want to populate 'col2' and leave 'col3' as it is we can apply the function specifically to that column:

[source,r]
----

> data.frame(col1 = c(1,2,3,4,5), col2  = c("a", NA, NA , "b", NA), col3 = c("A", NA, "B", NA, NA)) %>%
    mutate(col2 = na.locf(col2))
  col1 col2 col3
1    1    a    A
2    2    a <NA>
3    3    a    B
4    4    b <NA>
5    5    b <NA>
----

It's quite a neat function and certainly comes in helpful when cleaning up data sets which don't tend to be as uniform as you'd hope!