markhneedham/blog huggingface dataset:
Clojure: Anonymous functions using short notation and the 'ArityException Wrong number of args (0) passed to: PersistentVector'
Date: 2013-09-23 21:42:12

In the time I've spent playing around with Clojure one thing I've always got confused by is the error message you get when trying to return a vector using the anonymous function shorthand.

For example, if we want function which creates a vector with the values 1, 2, and the argument passed into the function we could write the following:

[source,lisp]
----

> ((fn [x] [1 2 x]) 6)
[1 2 6]
----

However, when I tried to convert it to the shorthand '#()' syntax I got the following exception:

[source,lisp]
----

> (#([1 2 %]) 6)
clojure.lang.ArityException: Wrong number of args (0) passed to: PersistentVector
                                      AFn.java:437 clojure.lang.AFn.throwArity
                                       AFn.java:35 clojure.lang.AFn.invoke
                                  NO_SOURCE_FILE:1 user/eval575[fn]
                                  NO_SOURCE_FILE:1 user/eval575
----

On previous occasions I've just stopped there and gone back to the long hand notation but this time I wanted to figure out why it didn't work as I expected.

I came across http://stackoverflow.com/questions/13204993/anonymous-function-shorthand[this StackOverflow post] which explained the way the shorthand gets expanded:

[source,lisp]
----

#() becomes (fn [arg1 arg2] (...))
----

which means that:

[source,lisp]
----

#(([1 2 %]) 6) becomes ((fn [arg] ([1 2 arg])) 6)
----

We are evaluating the vector +++<cite>+++[1 2 arg]+++</cite>+++ as a function but aren't passing any arguments to it. One way it can be used as a function is if we want to return a value at a specific index e.g.

[source,lisp]
----

> ([1 2 6] 2)
6
----

We don't want to evaluate a vector as a function, rather we want to return the vector using the shorthand syntax. To do that we need to find a function which will return the argument passed to it and then pass the vector to that function.

The +++<cite>+++http://clojuredocs.org/clojure_core/clojure.core/identity[identity]+++</cite>+++ function is one such function:

[source,lisp]
----

> (#(identity [1 2 %]) 6)
[1 2 6]
----

Or if we want to be more concise the +++<cite>+++http://clojuredocs.org/clojure_core/clojure.core/-%3E[thread-first (\->)]+++</cite>+++ works too:

[source,lisp]
----

> (#(-> [1 2 %]) 6)
[1 2 6]
----
cURL: POST/Upload multi part form
Date: 2013-09-23 22:16:29

I've been doing some work which involved uploading a couple of files from a HTML form and I wanted to check that the server side code was working by executing a cURL command rather than using the browser.

The form looks like this:

[source,html]
----

<form action="http://foobar.com" method="POST" enctype="multipart/form-data">
    <p>
        <label for="nodes">File 1:</label>
        <input type="file" name="file1" id="file1">
    </p>


    <p>
        <label for="relationships">File 2:</label>
        <input type="file" name="file2" id="file2">
    </p>


    <input type="submit" name="submit" value="Submit">
</form>
----

If we convert the POST request from the browser into a cURL equivalent we end up with the following:

[source,bash]
----

curl 'http://foobar.com' -H 'Origin: null' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Host: foobar.com:7474' -H 'Accept-Language: en-US,en;q=0.8' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/28.0.1500.95 Safari/537.36' -H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryMxYFIg6GFEIPAe6V' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Cache-Control: max-age=0' -H 'Cookie: splashShown1.6=1; undefined=0; _mkto_trk=id:773-GON-065&token:_mch-localhost-1373821432078-37666; JSESSIONID=123cbkxby1rtcj3dwipqzs7yu' -H 'Connection: keep-alive' --data-binary $'------WebKitFormBoundaryMxYFIg6GFEIPAe6V\r\nContent-Disposition: form-data; name="file1"; filename="file1.csv"\r\nContent-Type: text/csv\r\n\r\n\r\n------WebKitFormBoundaryMxYFIg6GFEIPAe6V\r\nContent-Disposition: form-data; name="file2"; filename="file2.csv"\r\nContent-Type: text/csv\r\n\r\n\r\n------WebKitFormBoundaryMxYFIg6GFEIPAe6V\r\nContent-Disposition: form-data; name="submit"\r\n\r\nSubmit\r\n------WebKitFormBoundaryMxYFIg6GFEIPAe6V--\r\n' --compressed
----

I tried executing this command but I couldn't quite get it to work, and in any case it seemed extremely complicated. Thankfully, I came across a http://curl.haxx.se/docs/httpscripting.html[cURL tutorial] which described a much simpler alternative which does work:

[source,bash]
----

curl --form file1=@file1.csv --form file2=@file2.csv --form press=submit http://foobar.com
----

I knew it couldn't be that complicated!
Clojure: Stripping all the whitespace
Date: 2013-09-22 18:54:47

When putting together data sets to play around with, one of the more boring tasks is stripping out characters that you're not interested in and more often than not those characters are white spaces.

Since I've been building data sets using Clojure I wanted to write a function that would do this for me.

I started out with the following string:

[source,lisp]
----

(def word " with a  little bit of space we can make it through the night  ")
----

which I wanted to format in such a way that there would be a maximum of one space between each word.

I start out by using the +++<cite>+++http://clojure.github.io/clojure/clojure.string-api.html#clojure.string/trim[trim]+++</cite>+++ function but that only removes white space from the beginning and end of a string:

[source,lisp]
----

> (clojure.string/trim word)
"with a  little bit of space we can make it through the night"
----

I wanted to get rid of the space in between 'a' and 'little' as well so I wrote the following code to split on a space and filter out any excess spaces that still remained before joining the words back together:

[source,lisp]
----

> (clojure.string/join " "
                       (filter #(not (clojure.string/blank? %))
                               (clojure.string/split word #" ")))
"with a little bit of space we can make it through the night"
----

I wanted to try and make it a bit easier to read by using the http://clojuredocs.org/clojure_core/clojure.core/-%3E%3E[thread last (\->>) macro] but that didn't work as well as I'd hoped because +++<cite>+++clojure.string/split+++</cite>+++ doesn't take the string in as its last parameter:

[source,lisp]
----

>  (->> (clojure.string/split word #" ")
   (filter #(not (clojure.string/blank? %)))
   (clojure.string/join " "))
"with a little bit of space we can make it through the night"
----

I worked around it by creating a specific function for splitting on a space:

[source,lisp]
----

(defn split-on-space [word]
  (clojure.string/split word #"\s"))
----

which means we can now chain everything together nicely:

[source,lisp]
----

>  (->> word
        split-on-space
        (filter #(not (clojure.string/blank? %)))
        (clojure.string/join " "))
"with a little bit of space we can make it through the night"
----

I couldn't find a cleaner way to do this but I'm sure there is one and my googling just isn't up to scratch so do let me know in the comments!
Clojure/Emacs/nrepl: Stacktrace-less error messages
Date: 2013-09-22 23:07:04

Ever since I started using the Emacs + nrepl combination to play around with Clojure I've been getting fairly non descript error messages whenever I pass the wrong parameters to a function.

For example if I try to update a non existent key in a form I get a Null Pointer Exception:

[source,lisp]
----

> (update-in {} [:mark] inc)
NullPointerException   clojure.lang.Numbers.ops (Numbers.java:942)
----

In this case it's clear that the hash doesn't have a key ':mark' so the function blows up. However, sometimes the functions are more complicated and this type of reduced stack trace isn't very helpful for working out where the problem lies.

I eventually came across https://groups.google.com/forum/#!msg/nrepl-el/x8vhEbckycY/93unZrJI1RkJ[a thread in the nrepl-el forum] where Tim King suggested that adding the following lines to the Emacs configuration file should sort things out:

_~/.emacs.d/init.el_

[source,lisp]
----

(setq nrepl-popup-stacktraces nil)
(setq nrepl-popup-stacktraces-in-repl t)
----

I added those two lines, restarted Emacs and after calling the function again got a much more detailed stack trace:

[source,lisp]
----

> (update-in {} [:mark] inc)

java.lang.NullPointerException:
                 Numbers.java:942 clojure.lang.Numbers.ops
                 Numbers.java:110 clojure.lang.Numbers.inc
                     core.clj:863 clojure.core/inc
                     AFn.java:161 clojure.lang.AFn.applyToHelper
                     AFn.java:151 clojure.lang.AFn.applyTo
                     core.clj:603 clojure.core/apply
                    core.clj:5472 clojure.core/update-in
                  RestFn.java:445 clojure.lang.RestFn.invoke
                 NO_SOURCE_FILE:1 user/eval9
...
----

From reading this stack trace we learn that the problem happens when the +++<cite>+++inc+++</cite>+++ function is called with a parameter of 'nil'. We'd see the same thing if we called it directly:

[source,lisp]
----

> (inc nil)

java.lang.NullPointerException:
                                  Numbers.java:942 clojure.lang.Numbers.ops
                                  Numbers.java:110 clojure.lang.Numbers.inc
                                  NO_SOURCE_FILE:1 user/eval14
...
----

Although Clojure error messages do baffle me at times, I hope things will be better now that I'll be able to see on which line the error occurred.
Clojure/Emacs/nrepl: Ctrl X + Ctrl E leads to 'FileNotFoundException Could not locate [...] on classpath'
Date: 2013-09-22 21:23:25

I've been playing around with Clojure using http://www.gnu.org/software/emacs/[Emacs] and https://github.com/clojure/tools.nrepl[nrepl] recently and my normal work flow is to write some code in Emacs and then have it evaluated in nrepl by typing +++<cite>+++Ctrl X + Ctrl E+++</cite>+++ at the end of the function.

I tried this once recently and got the following exception instead of a successful evaluation:

[source,text]
----

FileNotFoundException Could not locate ranking_algorithms/ranking__init.class or ranking_algorithms/ranking.clj on classpath: clojure.lang.RT.load (RT.java:432)
----

I was a bit surprised because I had nrepl running already (via +++<cite>+++(Meta + X) + Enter + nrepl-jack-in+++</cite>+++) and I'd only ever seen that exception refer to dependencies which weren't in my project.clj file at the time I launched nrepl.

I eventually came across http://stackoverflow.com/questions/15511840/emacs-clojure-compiling-error[this StackOverflow post] which suggested that you either launch nrepl using leiningen and then connect to it from Emacs or have your project.clj open when running +++<cite>+++(Meta + X) + Enter + nrepl-jack-in+++</cite>+++.

To launch nrepl from leiningen we'd run the following command from the terminal:

[source,bash]
----

$ lein repl
nREPL server started on port 52265
REPL-y 0.1.0-beta10
Clojure 1.4.0
    Exit: Control+D or (exit) or (quit)
Commands: (user/help)
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
          (user/sourcery function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
Examples from clojuredocs.org: [clojuredocs or cdoc]
          (user/clojuredocs name-here)
          (user/clojuredocs "ns-here" "name-here")
----

We can then connect to that nrepl server from Emacs by typing +++<cite>+++(Meta + X) + Enter + nrepl+++</cite>+++ which seems to work quite nicely.

To check the +++<cite>+++nrepl-jack-in+++</cite>+++ approach works when we've got project.clj open we need to first kill the existing server by typing +++<cite>+++(Meta + X) + Enter + nrepl-quit+++</cite>+++.

Now if we type +++<cite>+++(Meta + X) + Enter + nrepl-jack-in+++</cite>+++ our functions are evaluated correctly and all is well with the world again.
BT Internet: Non existent hosts mapping to 92.242.132.15
Date: 2013-08-17 21:13:27

We have a test in our code which checks for unresolvable hosts and it started failing for me because instead of throwing an UnknownHostException from the following call:

[source,java]
----

InetAddress.getByName( "host.that.is.invalid" )
----

I was getting back a valid although unreachable host. When I called ping it was easier to see what was going on:

[source,bash]
----

$ ping host.that.is.invalid
PING host.that.is.invalid (92.242.132.15): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
----

As you can see, that hostname is resolving to '92.242.132.15' which I thought was a bit weird but dig confirmed that this was happening:

[source,bash]
----

$ dig host.that.is.invalid

; <<>> DiG 9.8.3-P1 <<>> host.that.is.invalid
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 30043
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;host.that.is.invalid.		IN	A

;; ANSWER SECTION:
host.that.is.invalid.	300	IN	A	92.242.132.15
----

It turns out that http://community.bt.com/t5/BB-Speed-Connection-Issues/DNS-oddity/td-p/49653[BT have plugged into DNS searches] and if one fails it redirects you to one of their pages instead - something I hadn't noticed before.

The site they direct you to is www.webaddresshelp.bt.com which contains a list of sponsored results for the search term 'host.that.is.invalid' in this case.

[source,bash]
----

$ ping www.webaddresshelp.bt.com
PING www.webaddresshelp.bt.com (92.242.134.15): 56 data bytes
----

Luckily this can be disabled by going to link:[BT Web Address Help] and then choosing to http://preferences.webaddresshelp.bt.com/selfcare/preferences.cgi[disable BT Web Address Help].

If we then wait a little bit for our DNS cache to clear the ping works as expected:

[source,bash]
----

$ ping host.that.is.invalid
ping: cannot resolve host.that.is.invalid: Unknown host
----
Coding: Hack then revert
Date: 2013-08-19 23:13:04

For a long while my default approach when I came across a new code base that I wanted to change was to read all the code and try and understand how it all fitted together by sketching out flow of control diagrams.

Only after I'd done that would I start planning how I could make my changes.

This works reasonably well but it's quite time consuming and a couple of years ago a former colleague (I can't remember who!) showed me another technique which seems to be more effective.

Rather than trying to understand how all the code fits together we briefly skim it to get a general understanding but don't drill into the specifics.

Instead once we've got a general understanding we make changes to the code and then either run the application or run the tests to see if it works as we expected.

There'll often be a couple of cycles before we understand exactly what changes we need to make and I've found that *reverting the code after each attempt* works quite well.

When we change the same bit of code for the 2nd/3rd/4th time it takes a fraction of the time it did on the 1st occasion and we'll often spot improvements that we can make which we didn't notice before.

I'd recommend this as an exploratory tool if you haven't already tried it and as an added bonus it's much more fun than statically analysing code and trying to figure out how it's mean to work!
Neo4j's Graph Café London - 28th August 2013
Date: 2013-08-31 10:52:14

On Wednesday evening I attended an interesting spin on the monthly Neo4j meetup, where instead of the usual 'talk then go to the pub afterwards' format my colleagues https://twitter.com/rvanbruggen[Rik] and https://twitter.com/arturasver[Arturas] organised http://www.meetup.com/graphdb-london/events/123869482/[Graph Café] in the Doggetts Coat and Badge pub in Blackfriars.

The format was changed as well - the evening consisted of ~10 lightening talks which were spread out over about 3 hours, an approach Rik has used at http://blog.neo4j.org/2013/05/graph-cafes-starting-in-belgium-and.html[similar events] in Belgium and Holland earlier in the year.

In the gaps in between the talks people mingled with each other and shared tips/talked through the problems they were trying to solve using graphs.

There was a strong turn out and it was *much more interactive than a normal meet-up* where the main interaction comes from people asking the speaker questions. While there is a pub afterwards there's always a noticeable drop out from the talk so it was good to have everyone together chatting this time.

https://twitter.com/fgibson/status/372751710550106112[Frank Gibson] described what I thought was the coolest use of graphs of the evening. He's modelling different drugs, which medical conditions they treat and which other drugs they aren't compatible with. The next step is to bring that together with patients' medical records to help doctors make treatment recommendations.

As well as the talks, table clothes were laid out on the tables where people could sketch out the problems they were working on and get input from others. https://twitter.com/thobe[Tobias] went a bit meta and drew a graph about graph databases:

image::{{<siteurl>}}/uploads/2013/08/neograph.jpg[Neograph,600]

While it's often said that graphs are http://www.slideshare.net/slideshow/embed_code/12275060?startSlide=19[whiteboard friendly] I was still surprised at how effective this was. When someone was explaining what they were working on I found myself sketching out what I'd interpreted and then they'd join in and point out bits I'd misunderstood and bits they were thinking about changing.

Along those lines, my colleague http://www.apcjones.com/[Alistair] also demoed http://www.apcjones.com/arrows/[Arrows] - a Javascript library for drawing graphs which I use for most of the graph related diagrams on here.

image::{{<siteurl>}}/uploads/2013/08/outside.jpg[Outside,600]

Overall it was a fun meet up and now we need to try and work out how to keep the interactive aspect when it isn't 25 degrees outside and we're not in a pub overlooking http://www.stpauls.co.uk/[St Paul's].

Rik has put up https://plus.google.com/photos/112536463589645792207/albums/5917315641597676929?banner=pwa[a slide show of pictures] from the event and if you're interested in hosting your own similar event you should probably https://twitter.com/rvanbruggen[ping Rik] for some tips!
Python: for/list comprehensions and dictionaries
Date: 2013-08-13 22:59:52

I've been working through Coursera's https://class.coursera.org/matrix-001/class/index[Linear Algebra course] and since all of the exercises are in Python I've been playing around with it again.

One interesting thing I learnt is that you can construct dictionaries using a http://www.secnetix.de/olli/Python/list_comprehensions.hawk[list comprehension] type syntax.

For example, if we start with the following dictionaries:

[source,python]
----

>>> x = { "a": 1, "b":2 }
>>> y = {1: "mark", 2: "will"}
>>> x
{'a': 1, 'b': 2}
>>> y
{1: 'mark', 2: 'will'}
----

We might want to create a new dictionary which links from the keys in x to the values in y. In this case we work out the mapping by finding the key in y which corresponds with each value in x.

So the map we want to see at the end should look like this:

[source,text]
----

{"a": 'mark', "b": 'will'}
----

We can iterate over the keys/values of a dictionary by calling +++<cite>+++Dictionary#iteritems+++</cite>+++ like so:

[source,python]
----

>>> for key, value in x.iteritems():
...   print (key, value)
...
('a', 1)
('b', 2)
----

I thought I might be able to construct my new dictionary by converting this into a for comprehension:

[source,python]
----

>>> [key:value for key, value in x.iteritems()]
  File "<stdin>", line 1
    [key:value for key, value in x.iteritems()]
        ^
SyntaxError: invalid syntax
----

Unfortunately that didn't work but I came across http://stackoverflow.com/questions/3294889/iterating-over-dictionaries-for-loops-in-python[an interesting post] from which I learned that using curly brackets might do the trick.

[source,python]
----

>>> {key:value for key, value in x.iteritems()}
{'a': 1, 'b': 2}
>>> type({key:value for key, value in x.iteritems()})
<type 'dict'>
----

I wanted to make the final dictionary take a lookup into account which we can do like this:

[source,python]
----

>>> {key:y[value] for key, value in x.iteritems()}
{'a': 'mark', 'b': 'will'}
----

Apparently this is known as a http://www.python.org/dev/peps/pep-0274/[dictionary comprehension] and has http://stackoverflow.com/questions/7276511/are-there-dictionary-comprehensions-in-python-problem-with-function-returning[been in the language since version 2.7].

I'm sure this is old news to seasoned Python developers but I'd not come across it before so to me it's pretty neat!
Products & Infinite configurability
Date: 2013-08-22 22:11:35

One of the common feature requests on the http://www.thoughtworks.com/[ThoughtWorks] projects that I worked on was that the application we were working on should be almost infinitely configurable to cover potential future use cases.

My experience of attempting to do this was that you ended up with an extremely complicated code base and those future use cases often didn't come to fruition.

It therefore made more sense to *solve the problem at hand* and then make the code more configurable if/when the need arose.

Now that I'm working on a product and associated tools I'm trying to understand whether those rules of application development apply.

One thing which I think makes sense is the idea of http://en.wikipedia.org/wiki/Convention_over_configuration[convention over configuration], an approach that I became familiar with after working with Ruby/Rails in 2010/2011.

____
The phrase essentially means *a developer only needs to specify unconventional aspects of the application*.
____

Even if we do this I wonder if it goes far enough. The more things we make configurable the more complexity we add and the more opportunity for people to create themselves problems through misconfiguration.

Perhaps we should only make a few things configurable and have our application work out appropriate values for everything else.

There are a reasonable number of people using a product who don't have much interest in learning how to configure it. They *just want to use it to solve a problem they have* without having to think too much.

Although I haven't used it I'm told that http://www.azulsystems.com/products/zing/whatisit[Azul's Zing JVM] takes the minimal configuration approach by only requiring you to specify one parameter - the heap size - and it handles everything else for you.

Of course I'm still new to this so perhaps it still does make sense to default most things but allow power users full control in case their use case differs from the average one that the defaults were created for.

I'd be interested in hearing the opinions of people more experienced in this arena of which there are undoubtably many.