markhneedham/blog huggingface dataset:
Product Documentation: The receiver decides if it's successful
Date: 2013-07-28 16:18:38

One of the things I remember being taught while growing up is that in an interaction where somebody is something to someone else it's their responsibility to do so in a way that the receiver can understand.

Even if you think you've done a good job of explaining something, the receiver of the communication decides whether or not that's the case.

I'd always assumed that this advice made most sense in the context of a one to one conversation but recently I've realised that it also makes sense when thinking about product documentation.

Most products on the market have some sort of documentation available which is a good first step but I've still seen a couple of ways that users of the product struggle:

== It's too complicated

Frequently documentation is written by an expert of the product so the language they use is very technical and difficult for a novice to understand.

It's often worth running documentation past a non expert to get a sanity check but a tell tale sign that documentation may be too complicated is when you get a lot of questions about things that you believe it covers.

At this point we can often work out what has led to user confusion and then make the appropriate changes.

== People can't find it

Sometimes there's no problem with the documentation but people can't find it.

This might be because it's poorly titled or doesn't use the same language that users use when they are trying to solve a particular problem.

There's a tendency to believe that users should learn the language of the domain and while that's true to an extent, if we can work out what language they use it can reduce friction and lead to a better experience.

The other problem users encounter is where they find the documentation index but are then overwhelmed with information to the point where they're not sure what they should be clicking on.

This can be solved by focusing documentation on the simple case but having links through to more advanced topics for people who are interested.

== Further reading

Tom Preston Werner wrote a post a few years ago around this topic titled 'http://tom.preston-werner.com/2010/08/23/readme-driven-development.html[Readme Driven Development]' which is worth a read.

https://twitter.com/joewalnes[Joe Walnes] uses a similar approach on his projects - http://smoothiecharts.org/[Smoothie Charts] and http://webconnect.io/[WebConnect] are a couple of examples that I've come across.
Lessons from supporting production code
Date: 2013-07-22 22:37:28

Until I started working on the http://www.uswitch.com/[uSwitch] energy website around 8 months ago I had not really done any support of a production system so I learnt some interesting lessons in my time there.

== Look at the new code first

We had our application wired up to http://airbrakeapp.com/pages/home[Airbrake] so whenever a user did anything which resulted in an exception being thrown we received a report with the stack trace, environment variables and which page they were on.

When trying to work out what had happened I initially started from scratch and tried to work backwards from the source and create a scenario in my head of what they might have done to get that error.

After a few times of doing this it became clear that there was a reasonable chance that if a user was experiencing a problem it was probably because of some new code that we'd just introduced.

We therefore tweaked our bug hunting algorithm to initially check code that had been changed recently and only after ruling that out did we work back from first principles.

== It may never have worked

Sometimes it became clear that new code wasn't to blame but it seemed implausible that the error could have actually happened.

There was a tendency to assume that the user must be deliberately doing something to make the application break but it soon became clear that they had just managed to hit a code path that had not been hit before.

Even if you've done extensive testing on a system users still seem to find paths through the code that haven't failed previously so it seems best to just assume that is going to happen at some stage.

== Log all the things

As I mentioned earlier we were using a 3rd party service to collect errors and other helpful information which was really useful for helping us find the root cause of problems.

The type of logging that you need varies so for a product like http://www.neo4j.org/[neo4j] as well as logging exceptions we also log system information and memory settings.

Obviously I'm quite new to this type of work so I'm sure others will have useful bits of advice to share as well.
On "The fear of blogging about technical topics"
Date: 2013-07-22 23:47:28

My former colleague https://twitter.com/annejsimmons[Anne Simmons] recently wrote an interesting post in which she describes http://annejsimmons.com/2013/07/16/the-fear-of-blogging-about-technical-topics/[some of the reasons that she finds herself not wanting to write about technical topics.].

I wrote a post at the end of 2012 in which I explained some of the reasons why I think http://www.markhneedham.com/blog/2013/01/05/a-new-years-idea-share-what-you-learn/[writing about what you learn is a good idea] but Anne brought up some things I hadn't thought of which I think are worth addressing.

She's already described her own mantras to overcome these but I thought it'd still be interesting to share my experience as well:

== What do I know that the internet doesn't already?!

I've found that then posts I write in which I aggregate a bunch of information that I found from different places tend to be my most read posts.

A lot of the people that I've worked with (including me) when encountered with a stack trace will paste it straight into google to try and solve their problem.

I've had the same experience as Anne in spending ages trying to solve a problem and thinking it would be cool to save someone else (usually future me) from having to rediscover the solution in the future.

== Will people judge me about what knowledge I do have?/What if I'm wrong?!

In 5 years of writing I've only had a couple of times when people commented in what I thought was an unnecessary manner but the majority of feedback has been very positive.

Frequently people actually teach me something that I didn't know rather than criticising what I do know so for me writing has been a net gain.

A strange side effect is that people think I know much more than I do based on writing about things I've been working on.

== In summary

It can only be a good thing for the internet if more people write about the things that they're working on so I hope Anne keeps to her one post a month target!

I'll finish off with the http://www.catb.org/esr/faqs/hacker-howto.html[Eric Raymond] quote that I used on http://www.markhneedham.com/blog/2013/01/05/a-new-years-idea-share-what-you-learn/[my previous post about sharing what you learn]:

____
Creative brains are a valuable, limited resource. They shouldn't be wasted on re-inventing the wheel when there are so many fascinating new problems waiting out there. To behave like a hacker, you have to believe that the thinking time of other hackers is precious -- so much so that *it's almost a moral duty for you to share information, solve problems and then give the solutions away just so other hackers can solve new problems instead of having to perpetually re-address old ones*.
____
Clojure: Converting a string to a date
Date: 2013-09-20 07:00:01

I wanted to do some date manipulation in Clojure recently and figured that since https://github.com/seancorfield/clj-time[clj-time] is a wrapper around http://www.joda.org/joda-time/[Joda Time] it'd probably do the trick.

The first thing we need to do is add the dependency to our project file and then run +++<cite>+++lein reps+++</cite>+++ to pull down the appropriate JARs. The project file should look something like this:

_project.clj_

[source,lisp]
----

(defproject ranking-algorithms "0.1.0-SNAPSHOT"
  :license {:name "Eclipse Public License"
            :url "http://www.eclipse.org/legal/epl-v10.html"}
  :dependencies [[org.clojure/clojure "1.4.0"]
                 [clj-time "0.6.0"]])
----

Now let's load the clj-time.format namespace into the REPL since we know we'll be parsing dates:

[source,lisp]
----

> (require '(clj-time [format :as f]))
----

The string that I want to convert into a date looks like this:

[source,lisp]
----

(def string-date "18 September 2012")
----

The first thing we should do is check whether there is an existing formatter that we can use by evaluating the following function:

[source,lisp]
----

> (f/show-formatters)
...
:hour-minute                            06:45
:hour-minute-second                     06:45:22
:hour-minute-second-fraction            06:45:22.473
:hour-minute-second-ms                  06:45:22.473
:mysql                                  2013-09-20 06:45:22
:ordinal-date                           2013-263
:ordinal-date-time                      2013-263T06:45:22.473Z
:ordinal-date-time-no-ms                2013-263T06:45:22Z
:rfc822                                 Fri, 20 Sep 2013 06:45:22 +0000
...
----

There are a lot of different built in formatters but unfortunately I couldn't find one that exactly matched our date format so we'll have to write our own one.

For that we'll need to refresh our knowledge of http://docs.oracle.com/javase/1.4.2/docs/api/java/text/SimpleDateFormat.html[Java date formatting]:

image::{{<siteurl>}}/uploads/2013/09/2013-09-20_07-48-52.png[2013 09 20 07 48 52,600]

We end up with the following formatter:

[source,lisp]
----

> (f/parse (f/formatter "dd MMM YYYY") string-date)
#<DateTime 2012-09-18T00:00:00.000Z>
----

It took me much longer than it should have to remember that 'MMM' is the pattern to match a short form of a month but it's just the same as what we'd have to do in Java but with some neat wrapper functions.
Clojure: Merge two maps but only keep the keys of one of them
Date: 2013-09-17 01:03:37

I've http://www.markhneedham.com/blog/2013/09/17/clojure-updating-keys-in-a-map/[been playing around with Clojure maps] recently and I wanted to merge two maps of rankings where the rankings in the second map overrode those in the first while only keeping the teams from the first map.

The +++<cite>+++http://clojuredocs.org/clojure_core/clojure.core/merge[merge]+++</cite>+++ function overrides keys in earlier maps but also adds keys that only appear in later maps. For example, if we merge the following maps:

[source,lisp]
----

> (merge {"Man. United" 1500 "Man. City" 1400} {"Man. United" 1550 "Arsenal" 1450})
{"Arsenal" 1450, "Man. United" 1550, "Man. City" 1400}
----

we get back all 3 teams but I wanted a function which only returned 'Man. United' and 'Man. City' since those keys appear in the first map and 'Arsenal' doesn't.

I http://stackoverflow.com/questions/2753874/how-to-filter-a-persistent-map-in-clojure[wrote the following function]:

[source,lisp]
----

(defn merge-rankings [initial-rankings override-rankings]
  (merge initial-rankings
         (into {} (filter #(contains? initial-rankings (key %)) override-rankings))))
----

If we call that we get the desired result:

[source,lisp]
----

> (merge-rankings {"Man. United" 1500 "Man. City" 1400} {"Man. United" 1550 "Arsenal" 1450})
{"Man. United" 1550, "Man. City" 1400}
----

An alternative version of that function could use +++<cite>+++http://clojuredocs.org/clojure_core/clojure.core/select-keys[select-keys]+++</cite>+++ like so:

[source,lisp]
----

(defn merge-rankings [initial-rankings override-rankings]
  (select-keys (merge initial-rankings override-rankings) (map key initial-rankings)))
----

bitemyapp points out in the comments that we can go even further and use the +++<cite>+++http://clojuredocs.org/clojure_core/clojure.core/keys[keys]+++</cite>+++ function instead of +++<cite>+++map key+++</cite>+++, like so:

[source,lisp]
----

(defn merge-rankings [initial-rankings override-rankings]
  (select-keys (merge initial-rankings override-rankings) (keys initial-rankings)))
----

Now let's generify the function so it would make sense in the context of any maps, not just ranking related ones:

[source,lisp]
----

(defn merge-keep-left [left right]
  (select-keys (merge left right) (keys left)))
----
Clojure: Updating keys in a map
Date: 2013-09-17 00:24:48

I've been playing with Clojure over the last few weeks and as a result I've been using a lot of maps to represent the data.

For example if we have the following map of teams to http://www.glicko.net/glicko/glicko.pdf[Glicko] ratings and ratings deviations:

[source,lisp]
----

(def teams { "Man. United" {:points 1500 :rd 350}
             "Man. City"   {:points 1450 :rd 300} })
----

We might want to increase Man. United's points score by one for which we could use the +++<cite>+++http://clojuredocs.org/clojure_core/1.2.0/clojure.core/update-in[update-in]+++</cite>+++ function:

[source,lisp]
----

> (update-in teams ["Man. United" :points] inc)
{"Man. United" {:points 1501, :rd 350}, "Man. City" {:points 1450, :rd 300}}
----

The 2nd argument to +++<cite>+++update-in+++</cite>+++ is a nested associative structure i.e. a sequence of keys into the map in this instance.

If we wanted to reset Man. United's points score we could use +++<cite>+++http://clojuredocs.org/clojure_core/clojure.core/assoc-in[assoc-in]+++</cite>+++:

[source,lisp]
----

> (assoc-in teams ["Man. United" :points] 1)
{"Man. United" {:points 1, :rd 350}, "Man. City" {:points 1450, :rd 300}}
----

If we want to update multiple keys at once then we can chain them using the http://clojuredocs.org/clojure_core/clojure.core/-%3E[\->] (thread first) macro:

[source,lisp]
----

(-> teams
    (assoc-in ["Man. United" :points] 1600)
    (assoc-in ["Man. United" :rd] 200))
{"Man. United" {:points 1600, :rd 200}, "Man. City" {:points 1450, :rd 300}}
----

If instead of replacing just one part of the value we want to replace the whole entry we could use +++<cite>+++http://clojuredocs.org/clojure_core/clojure.core/assoc[assoc]+++</cite>+++ instead:

[source,lisp]
----

> (assoc teams "Man. United" {:points 1600 :rd 300})
{"Man. United" {:points 1600, :rd 300}, "Man. City" {:points 1450, :rd 300}}
----

+++<cite>+++assoc+++</cite>+++ can also be used to add a new key/value to the map. e.g.

[source,lisp]
----

> (assoc teams "Arsenal" {:points 1500 :rd 330})
{"Man. United" {:points 1500, :rd 350}, "Arsenal" {:points 1500, :rd 330}, "Man. City" {:points 1450, :rd 300}}
----

+++<cite>+++http://clojuredocs.org/clojure_core/clojure.core/dissoc[dissoc]+++</cite>+++ plays the opposite role and returns a new map without the specified keys:

[source,lisp]
----

> (dissoc teams "Man. United" "Man. City")
{}
----

And those are all the map based functions I've played around with so far\...
Clojure: See every step of a reduce
Date: 2013-09-19 23:57:49

Last year I wrote about http://www.markhneedham.com/blog/2012/02/25/haskell-viewing-the-steps-of-a-reduce/[a Haskell function called +++<cite>+++scanl+++</cite>+++] which returned the intermediate steps of a fold over a collection and last week I realised that I needed a similar function in Clojure to analyse a reduce I'd written.

A simple reduce which adds together the numbers 1-10 would look like this:

[source,lisp]
----

> (reduce + 0 (range 1 11))
55
----

If we want to see the intermediate values of this function called then instead of using +++<cite>+++http://clojuredocs.org/clojure_core/1.3.0/clojure.core/reduce[reduce]+++</cite>+++ there's a function called +++<cite>+++http://clojuredocs.org/clojure_core/1.3.0/clojure.core/reductions[reductions]+++</cite>+++ which gives us exactly what we want:

[source,lisp]
----

> (reductions + 0 (range 1 11))
(0 1 3 6 10 15 21 28 36 45 55)
----

I found this function especially useful when analysing http://www.markhneedham.com/blog/2013/09/14/glicko-rating-system-a-simple-example-using-clojure/[my implementation of the Glicko ranking algorithm] to work out whether a team's ranking was being updated correctly after a round of matches.

I initially thought the +++<cite>+++reductions+++</cite>+++ function was only useful as a debugging tool and that you'd always end up changing your code back to use +++<cite>+++reduce+++</cite>+++ after you'd solved the problem but I realise I was mistaken.

As part of http://www.markhneedham.com/blog/2013/09/14/glicko-rating-system-a-simple-example-using-clojure/[my implementation of the Glicko algorithm] I wrote a bit of code that applied a reduce across a collection of football seasons and initially just returned the final ranking of each team:

[source,lisp]
----

(def initial-team-rankings { "Man Utd" {:points 1200} "Man City" {:points 1300}})

(defn update-team-rankings [teams year]
  (reduce (fn [ts [team _]] (update-in ts [team :points] inc)) teams teams))
----

[source,lisp]
----

> (reduce update-team-rankings initial-team-rankings (range 2004 2013))
{"Man City" {:points 1309}, "Man Utd" {:points 1209}}
----

I realised it would actually be quite interesting to see the rankings after each season for which +++<cite>+++reductions+++</cite>+++ comes in quite handy.

For example if we want to find the rankings after 3 seasons we could write the following code:

[source,lisp]
----

> (nth (reductions update-team-rankings initial-team-rankings (range 2004 2013)) 3)
{"Man City" {:points 1303}, "Man Utd" {:points 1203}}
----

Or we could join the result back onto our collection of years and create a map so we can look up the year more easily:

[source,lisp]
----

(def final-rankings
  (zipmap (range 2003 2013) (reductions update-team-rankings initial-team-rankings (range 2004 2013))))
----

[source,lisp]
----

> (get final-rankings 2006)
{"Man City" {:points 1303}, "Man Utd" {:points 1203}}
----
Clojure: Writing JSON to a file/reading JSON from a file
Date: 2013-09-26 07:47:34

A few weeks ago I described how I'd http://www.markhneedham.com/blog/2013/08/26/clojureenlive-screen-scraping-a-html-file-from-disk/[scraped football matches using Clojure's Enlive], and the next step after translating the HTML representation into a Clojure map was to save it as a JSON document.

I decided to follow a two step process to achieve this:

* Convert hash to JSON string
* Write JSON string to file

I imagine there's probably a way to convert the hash to a stream and pipe that into a file but my JSON document isn't very large so I think this way is ok for now.

+++<cite>+++https://github.com/clojure/data.json[data.json]+++</cite>+++ seems to be the way to go to convert a Hash to a JSON string and I had the following code:

[source,lisp]
----

> (require '[clojure.data.json :as json])
nil

> (json/write-str { :key1 "val1" :key2 "val2" })
"{\"key2\":\"val2\",\"key1\":\"val1\"}"
----

The next step was to write that into a file and http://stackoverflow.com/questions/7756909/in-clojure-1-3-how-to-read-and-write-a-file[this StackOverflow post describes a couple of ways that we can do this]:

[source,lisp]
----

> (use 'clojure.java.io)
> (with-open [wrtr (writer "/tmp/test.json")]
    (.write wrtr (json/write-str {:key1 "val1" :key2 "val2"})))
----

or

[source,lisp]
----

> (spit "/tmp/test.json" (json/write-str {:key1 "val1" :key2 "val2"}))
----

Now I wanted to read the file back into a hash and I started with the following:

[source,lisp]
----

> (json/read-str (slurp "/tmp/test.json"))
{"key2" "val2", "key1" "val1"}
----

That's not bad but I wanted the keys to be what I know as symbols (e.g. ':key1') from Ruby land. I re-learnt that this is called a keyword in Clojure.

Since I'm not very good at reading the documentation I wrote a function to convert all the keys in a map from strings to keywords:

[source,lisp]
----

> (defn string-keys-to-symbols [map]
    (reduce #(assoc %1 (-> (key %2) keyword) (val %2)) {} map))

> (string-keys-to-symbols (json/read-str (slurp "/tmp/test.json")))
{:key1 "val1", :key2 "val2"}
----

What I should have done is pass the +++<cite>+++http://clojuredocs.org/clojure_core/clojure.core/keyword[keyword]+++</cite>+++ function as an argument to +++<cite>+++read-str+++</cite>+++ instead:

[source,lisp]
----

> (json/read-str (slurp "/tmp/test.json") :key-fn keyword)
{:key2 "val2", :key1 "val1"}
----

Simple!
Clojure: Writing JSON to a file - "Exception Don't know how to write JSON of class org.joda.time.DateTime"
Date: 2013-09-26 19:11:29

As I mentioned in an earlier post I've been transforming http://www.markhneedham.com/blog/2013/09/26/clojure-writing-json-to-a-filereading-json-from-a-file/[Clojure hash's into JSON strings] using https://github.com/clojure/data.json[data.json] but ran into trouble while trying to parse a hash which contained a http://www.joda.org/joda-time/[Joda Time] DateTime instance.

The date in question was constructed like this:

[source,lisp]
----

(ns json-date-example
  (:require [clj-time.format :as f])
  (:require [clojure.data.json :as json]))

(defn as-date [date-field]
  (f/parse (f/formatter "dd MMM YYYY") date-field ))

(def my-date
  (as-date "18 Mar 2012"))
----

And when I tried to convert a hash containing that object into a string I got the following exception:

[source,lisp]
----

> (json/write-str {:date my-date)})

java.lang.Exception: Don't know how to write JSON of class org.joda.time.DateTime
 at clojure.data.json$write_generic.invoke (json.clj:367)
    clojure.data.json$eval2818$fn__2819$G__2809__2826.invoke (json.clj:284)
    clojure.data.json$write_object.invoke (json.clj:333)
    clojure.data.json$eval2818$fn__2819$G__2809__2826.invoke (json.clj:284)
    clojure.data.json$write.doInvoke (json.clj:450)
    clojure.lang.RestFn.invoke (RestFn.java:425)
----

Luckily it's quite easy to get around this by passing a function to +++<cite>+++write-str+++</cite>+++ that converts the DateTime into a string representation before writing that part of the hash to a string.

The function looks like this:

[source,lisp]
----

(defn as-date-string [date]
  (f/unparse (f/formatter "dd MMM YYYY") date))

(defn date-aware-value-writer [key value]
  (if (= key :date) (as-date-string value) value))
----

And we make use of the writer like so:

[source,lisp]
----

> (json/write-str {:date my-date} :value-fn date-aware-value-writer)
"{\"date\":\"18 Mar 2012\"}"
----

If we want to read that string back again and reify our date we create a reader function which converts a string into a DateTime. The +++<cite>+++as-date+++</cite>+++ function from the beginning of this post does exactly what we want so we'll use that:

[source,lisp]
----

(defn date-aware-value-reader [key value]
  (if (= key :date) (as-date value) value))
----

We can then pass the reader as an argument to +++<cite>+++read-str+++</cite>+++:

[source,lisp]
----

> (json/read-str "{\"date\":\"18 Mar 2012\"}" :value-fn date-aware-value-reader :key-fn keyword)
{:date #<DateTime 2012-03-18T00:00:00.000Z>}
----