markhneedham/blog huggingface dataset:
Scala: Option.isDefined as the new null check
Date: 2011-11-01 00:58:45

One cool thing about using Scala on my current project is that we don't have nulls anywhere in our code, instead when something may or may not be there we make use of the Option type.

Unfortunately what we've (heavily contributed by me) ended up with in our code base is repeated use of the +++<cite>+++http://www.scala-lang.org/api/rc/scala/Option.html[isDefined]+++</cite>+++ method whenever we want to make a decision depending on whether or not the option is populated.

For example the following is quite common:

[source,scala]
----

case class Foo(val bar:String)
val foo : Option[Foo] = Some(Foo("mark"))
----

[source,scala]
----

> val bar = if(foo.isDefined) Some(foo.get.bar) else None
bar: Option[String] = Some(mark)
----

We can actually get rid of the if statement by making use of +++<cite>+++collect+++</cite>+++ instead:

[source,scala]
----

> val bar = foo.collect { case f => f.bar }
bar: Option[String] = Some(mark)
----

And if foo is +++<cite>+++None+++</cite>+++:

[source,scala]
----

> val foo : Option[Foo] = None
> val bar = foo.collect { case f => f.bar }
bar: Option[String] = None
----

The code is now simpler and as long as you understand +++<cite>+++collect+++</cite>+++ then it's easier to understand as well.

Another quite common example would be something like this:

[source,scala]
----

case class Foo(val bar:Option[String])
----

[source,scala]
----

> val foos = List(Foo(Some("mark")), Foo(None), Foo(Some("needham")))
foos: List[Foo] = List(Foo(Some(mark)), Foo(None), Foo(Some(needham)))
----

[source,scala]
----

> foos.filter(_.bar.isDefined).map(_.bar.get + " awesome")
res23: List[java.lang.String] = List(mark awesome, needham awesome)
----

Which we can simplify down to:

[source,scala]
----

foos.collect { case Foo(Some(bar)) => bar + " awesome" }
----

When I was playing around with F# a couple of years ago I learnt that wherever possible I should try and keep chaining functions together rather than breaking the code up into conditionals and I think the same applies here.

There are loads of methods available on +++<cite>+++http://www.scala-lang.org/api/rc/scala/collection/TraversableLike.html[TraversableLike]+++</cite>+++ to help us achieve this.
Scala: Which implicit conversion is being used?
Date: 2011-11-06 21:25:06

Last week my colleague http://twitter.com/patforna[Pat] created a method which had a parameter which he wanted to make optional so that consumers of the API wouldn't have to provide it if they didn't want to.

We ended up making the method take in an implicit value such that the method signature looked a bit like this:

[source,scala]
----

def foo[T](implicit blah:(String => T)) = {
  println(blah("mark"))
  "foo"
}
----

We can call foo with or without an argument:

[source,scala]
----

scala> foo { x => x + " Needham" }
mark Needham
res16: java.lang.String = foo
----

[source,scala]
----

scala> foo
mark
res17: java.lang.String = foo
----

In the second case it seems like the function is defaulting to an identity function of some sorts since the same value we pass to it is getting printed out.

We figured that it was probably using one of the implicit conversions in +++<cite>+++Predef+++</cite>+++ but weren't sure which one.

I asked about this on the Scala IRC channel and Heikki Vesalainen suggested running +++<cite>+++scala+++</cite>+++ with the '-print' flag to work it out.

[source,scala]
----

scala -print
----

The output is pretty verbose but having defined foo as above this is some of the output we get when calling it:

[source,scala]
----

scala> foo
[[syntax trees at end of cleanup]]// Scala source: <console>
package $line2 {
  final object $read extends java.lang.Object with ScalaObject {
    def this(): object $line2.$read = {
      $read.super.this();
      ()
    }
  };
  final object $read$$iw$$iw extends java.lang.Object with ScalaObject {
    private[this] val res0: java.lang.String = _;
    <stable> <accessor> def res0(): java.lang.String = $read$$iw$$iw.this.res0;
    def this(): object $line2.$read$$iw$$iw = {
      $read$$iw$$iw.super.this();
      $read$$iw$$iw.this.res0 = $line1.$read$$iw$$iw.foo(<strong>scala.this.Predef.conforms()</strong>);
      ()
    }
  };
  final object $read$$iw extends java.lang.Object with ScalaObject {
    def this(): object $line2.$read$$iw = {
      $read$$iw.super.this();
      ()
    }
  }
}
----

I've highlighted the call to +++<cite>+++Predef.conforms()+++</cite>+++ which is the implicit conversion that's been substituted into 'foo'.

It's defined like so:

http://www.scala-lang.org/api/current/index.html#scala.Predef$$$less$colon$less[Predef.scala]

[source,scala]
----

implicit def conforms[A]: A <:< A = new (A <:< A) { def apply(x: A) = x }
----

I'm not sure where that would be legitimately used but the comments just above it suggest the following:

[source,text]
----

An instance of `A <:< B` witnesses that `A` is a subtype of `B`.
----

This is probably a misuse of implcits and we intend to replace the implicit in our code with a default function value but it was interesting investigating where the implicit had come from!
Scala: scala.xml.SpecialNode: StackOverFlowError
Date: 2011-11-15 00:26:46

We have some code in our application where we parse reasonably complex XML structures and then sometimes choose to get rid of certain elements from the structure.

When we wanted to get rid of an element we replaced that element with a +++<cite>+++http://www.scala-lang.org/api/current/scala/xml/SpecialNode.html[SpecialNode]+++</cite>+++:

[source,scala]
----

val emptyNode = new scala.xml.SpecialNode() {
  def buildString(sb:StringBuilder) = new StringBuilder()
  def label = null
}
----

Unfortunately when you call #text on the node it results in the following exception which we only found out today:

[source,scala]
----

> emptyNode.text

java.lang.StackOverflowError
	at scala.xml.NodeSeq$$anonfun$text$1.apply(NodeSeq.scala:152)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:194)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:194)
	at scala.collection.Iterator$class.foreach(Iterator.scala:652)
	at scala.collection.LinearSeqLike$$anon$1.foreach(LinearSeqLike.scala:50)
	at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
	at scala.xml.NodeSeq.foreach(NodeSeq.scala:43)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:194)
	at scala.xml.NodeSeq.map(NodeSeq.scala:43)
	at scala.xml.NodeSeq.text(NodeSeq.scala:152)
	at scala.xml.Node.text(Node.scala:200)
----

The way to get around that problem is to override the text method so it returns empty:

[source,scala]
----

val emptyNode = new scala.xml.SpecialNode() {
  def buildString(sb:StringBuilder) = new StringBuilder()
  def label = null
  override def text = ""
}
----

[source,scala]
----

> emptyNode.text
res1: String = ""
----

It took a seriously long time for us to track down what was going on and that bit of code wasn't unit tested.

#fail
Git: Getting the history of a deleted file
Date: 2011-10-04 22:33:09

We recently wanted to get the Git history of a file which we knew existed but had now been deleted so we could find out what had happened to it.

Using a simple +++<cite>+++git log+++</cite>+++ didn't work:

[source,text]
----

git log deletedFile.txt
fatal: ambiguous argument 'deletedFile.txt': unknown revision or path not in the working tree.
----

We eventually came across http://feeding.cloud.geek.nz/2010/07/querying-deleted-content-in-git.html[Francois Marier's blog post] which points out that you need to use the following command instead:

[source,text]
----

git log -- deletedFile.txt
----

I've tried reading through the man page but I'm still not entirely sure what the distinction between using +++<cite>+++--+++</cite>+++ and not using it is supposed to be.

If someone could explain it that'd be cool...
Getting stuck and agile software teams
Date: 2011-10-20 22:09:31

I  came across an interesting set of posts by Jeff Wofford where he talks about http://www.jeffwofford.com/?p=835[programmers getting] http://www.jeffwofford.com/?p=838[stuck] and it made me think that, despite its faults, agile software development does have some useful practices for stopping us getting stuck for too long.

Many of the examples that Jeff describes sound like http://sethgodin.typepad.com/seths_blog/2005/03/dont_shave_that.html[yak shaving] to me which is part of what makes programming fun but doesn't always correlate to adding value to the product that you're building.

Although I wrote about some of the http://www.markhneedham.com/blog/2011/09/06/pair-programming-the-disadvantages-of-100-pairing/[disadvantages of pair programming] a while ago it is actually a very useful practice for ensuring that we don't get stuck.

We're much less likely to go off down a rabbit hole trying to solve some interesting but unrelated problem if we have to try and convince someone else to come along on that journey.

On most teams that I've worked on at least a reasonable percentage of the team is *co-located* so there's almost certainly going to be someone sitting nearby who will be able to help.

If that isn't enough, we tend to have a *very visible story wall* of what everyone's working on right next to the work space and it become pretty obvious when something has been stuck in one of the columns for a long time.

Another team member is bound to point that out and if they don't then the *standup* at the beginning of the day provides a good opportunity to see if anyone else on the team has a way around the problem you're working on.

It also provides an opportunity to find out whether the problem you're trying to solve is actually worth solving or not by talking to the product owner/one of the business analysts.

For the types of problems that I work on more often than not it isn't vital to solve a lot of problems that we think we need to and the product owner would much rather we just parked it and work on something else that is valuable to them.

Jeff goes on to http://www.jeffwofford.com/?p=843[describe some other more general ways of getting unstuck] but the above are some which might not be available to us with a less collaborative approach.
git: Only pushing some changes from local repository
Date: 2011-10-20 06:50:01

Something that we want to do reasonable frequently on my current project is to push some changes which have been committed to our local repository to master but not all of them.

For example we might end up with 3 changes we haven't pushed:

[source,text]
----

>> ~/github/local$ git status
# On branch master
# Your branch is ahead of 'origin/master' by 3 commits.
#
nothing to commit (working directory clean)
----

[source,text]
----

>> ~/github/local$ git hist
* bb7b139 Thu, 20 Oct 2011 07:37:11 +0100 | mark: one last time (HEAD, master) [Mark Needham]
* 1cef99a Thu, 20 Oct 2011 07:36:35 +0100 | mark:another new line [Mark Needham]
* 850e105 Thu, 20 Oct 2011 07:36:01 +0100 | mark: new line [Mark Needham]
* 2b25622 Thu, 20 Oct 2011 07:32:43 +0100 | mark: adding file for first time (origin/master) [Mark Needham]
----

And we only want to push the commit with hash +++<cite>+++850e105+++</cite>+++ for example.

The approach which my colleague https://github.com/uday-rayala[Uday] showed us is to first take a temporary branch of the current state.

[source,text]
----

>> ~/github/local$ git checkout -b temp-branch
Switched to a new branch 'temp-branch'
----

Then immediately switch back to +++<cite>+++master+++</cite>+++ and 'get rid' of the last two changes from there:

[source,text]
----

>> ~/github/local$ git checkout master
Switched to branch 'master'
Your branch is ahead of 'origin/master' by 3 commits.
----

[source,text]
----

>> ~/github/local$ git reset HEAD~2 --hard
HEAD is now at 850e105 mark: new line
----

We can then push just that change:

[source,text]
----

>> ~/github/local$ git push
Counting objects: 5, done.
Writing objects: 100% (3/3), 257 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
To /Users/mneedham/github/remote
   2b25622..850e105  master -> master
----

And merge the temporary branch back in again so we're back where we were before:

[source,text]
----

>> ~/github/local$ git merge temp-branch
Updating 850e105..bb7b139
Fast-forward
 foo.txt |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
----

[source,text]
----

>> ~/github/local$ git hist
* bb7b139 Thu, 20 Oct 2011 07:37:11 +0100 | mark: one last time (HEAD, temp-branch, master) [Mark Needham]
* 1cef99a Thu, 20 Oct 2011 07:36:35 +0100 | mark:another new line [Mark Needham]
* 850e105 Thu, 20 Oct 2011 07:36:01 +0100 | mark: new line (origin/master) [Mark Needham]
* 2b25622 Thu, 20 Oct 2011 07:32:43 +0100 | mark: adding file for first time [Mark Needham]
----

[source,text]
----

>> ~/github/local$ git status
# On branch master
# Your branch is ahead of 'origin/master' by 2 commits.
#
nothing to commit (working directory clean)
----

And finally we delete the temporary branch:

[source,text]
----

>> ~/github/local$ git branch -d temp-branch
Deleted branch temp-branch (was bb7b139).
----

We can achieve the same thing without creating the branch and just cherry picking the commits back again after we've pushed our changes but this seems approach seems quicker.
Unix: Some useful tools
Date: 2011-10-17 22:58:50

On my current project we regularly use a few Unix tools which aren't on the standard installation so I thought I'd collate them here so I don't forget about them in the future.

== ghex

We suspected we'd ended up with some rogue characters in a file that we weren't able to detect in our normal text editor recently and wanted to view the byte by byte representation of the file to check it out.

We came across +++<cite>+++http://live.gnome.org/Ghex[ghex]+++</cite>+++ which seems to be a pretty decent tool for allowing us to do this.

[source,text]
----

sudo port install ghex
----

[source,text]
----

ghex2 ourFile.jade
----

== axel

+++<cite>+++http://axel.alioth.debian.org/[axel]+++</cite>+++ is a download accelerator and lets us send multiple http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35[partial/range requests] to download parts of a file before putting it back together at the end.

We found this quite useful when I was working in India to download files from the US over VPN. scp was painfully slow so we used to set up a simple HTTP server on the US server and then use axel to grab the file.

Some servers don't support range requests but a reasonable number of them seem to.

[source,text]
----

sudo port install axel
----

[source,text]
----

axel -a http://www.cs.cmu.edu/~dga/papers/andersen-phd-thesis.pdf
----

== ack

The man page claims the following:

____
Ack is designed as a replacement for 99% of the uses of grep.
____

It worked reasonably well for replacing the following grep command:

[source,text]
----

grep -iR "searchTerm" .
----

One of the cool things is that by default it doesn't search in binary files whereas grep does. I have noticed that it sometimes doesn't pick up search terms in files which grep would match and I'm not entirely sure why.

[source,text]
----

sudo port install p5-app-ack
----

[source,text]
----

ack "something"
----

I'm sure there are plenty of other cool tools about so if you know of any let me know!
Unix: Getting the page count of a linearized PDF
Date: 2011-10-09 11:34:04

We were doing some work last week to rasterize a PDF document into a sequence of images and wanted to get a rough idea of how many pages we'd be dealing with if we created an image per page.

The PDFs we're dealing with are http://partners.adobe.com/public/developer/en/pdf/PDFReference.pdf[linearized] since they're available for viewing on the web:

____
A LINEARIZED PDF FILE is one that has been organized in a special way to enable efﬁcient incremental access in a network environment. The ﬁle is valid PDF in all respects, and is compatible with all existing viewers and other PDF applications. Enhanced viewer applications can recognize that a PDF ﬁle has been linearized and can take advantage of that organization (as well as added "`hint`" information) to enhance viewing performance.
____

The neat thing about this is it means that the document has meta data detailing the number of pages it contains:

____
Part 2: Linearization parameter dictionary +
43 0 obj +
<< /Linearized 1.0 % Version +
/L 54567 % File length +
/H [475 598] % Primary hint stream offset and length (part 5) +
/O 45 % Object number of ﬁrst page's page object (part 6) +
/E 5437 % Offset of end of ﬁrst page +
*/N 11 % Number of pages in document* +
/T 52786 % Offset of ﬁrst entry in main cross-reference table (part 11) +
>> +
endobj
____

By making use of the +++<cite>+++http://en.wikipedia.org/wiki/Strings_(Unix)[strings]+++</cite>+++ command http://duncan-cragg.org/blog/[Duncan] and I hacked together a little script that lets us grab the number of pages in http://www.rand.org/pubs/commercial_books/CB149-1.html[The Games of Strategy] http://www.rand.org/content/dam/rand/pubs/commercial_books/2007/RAND_CB149-1.pdf[PDF] or any other linearized PDF:

[source,text]
----

strings RAND_CB149-1.pdf |
awk '/Linearized/ { inmeta = 1; } match($0, /\/N [0-9]+/) { if(inmeta) print substr( $0, RSTART, RLENGTH ); exit;}' |
cut -d" " -f2
----

It seems much more difficult to find the count if the document hasn't been linearized but we didn't need to solve that problem for the moment!
Working with external identifiers
Date: 2011-10-31 22:58:29

As part of the ingestion process for our application we import XML documents and corresponding PDFs into a database and onto the file system respectively.

image::{{<siteurl>}}/uploads/2011/10/externalid.jpg[Externalid,261]

Since the user needs to be able to search for documents by the +++<cite>+++userFacingId+++</cite>+++ we reference it by that identifier in the database and the web application.

Each document also has an external identifier and we use this to identify the PDFs on the file system.

We can't use the raw  +++<cite>+++userFacingId+++</cite>+++ to do this because there are some documents which have the same ID when we import them.

Most of the time we only need to care about the +++<cite>+++userFacingId+++</cite>+++ in the web application but when the user wants to download a PDF we need to map from the +++<cite>+++userFacingId+++</cite>+++ to the </cite>externalId</cite> so we can locate the file on the file system.

The first implementation of this code involved some mapping code in the web application from which we constructed an +++<cite>+++externalId+++</cite>+++ from a given +++<cite>+++userFacingId+++</cite>+++.

Unfortunately this logic drifted into a few different places and it started to become really difficult to tell whether we were dealing with a +++<cite>+++userFacingId+++</cite>+++ or an +++<cite>+++externalId+++</cite>+++.

We wanted to try and isolate the translation logic into one place on the edge of the system but http://twitter.com/patforna[Pat] pointed out that it would actually be simpler if we never had to care about the +++<cite>+++externalId+++</cite>+++ in our code.

We changed the ingestion process to add the +++<cite>+++externalId+++</cite>+++ to each document so that we'd be able to get hold of it when we needed to.

image::{{<siteurl>}}/uploads/2011/10/lookup.jpg[Lookup,299]

We had to change the design of the code so that whenever the user wants to download a PDF (for example) we make a call to the database by the +++<cite>+++userFacingId+++</cite>+++ to look up the +++<cite>+++externalId+++</cite>+++.

The disadvantage of the approach is that we're making an extra (internal) network call to look up the ID but it's the type of code that should be easily cacheable if it becomes a performance problem so it should be fine.

I think this approach is much better than having potentially flawed translation logic in the application.
Canonical Identifiers
Date: 2011-10-30 22:32:16

http://duncan-cragg.org/blog/[Duncan] and I had an interesting problem recently where we had to make it possible to search within an 'item' to find possible sub items that exist inside it.

The http://en.wikipedia.org/wiki/Uniform_Resource_Identifier[URI] for the item was something like this:

[source,text]
----

/items/234
----

Let's say Item 234 contains the following sub items:

* Mark
* duncan

We have a search box on the page which allows us to type in the name of a sub item and go the sub item's page if it exists or see an error message if it doesn't.

If the user types in the sub item name exactly right then there's no problem:

[source,text]
----

items/234?subItem=Mark
----

redirects to:

[source,text]
----

items/234/subItem/Mark
----

It becomes more interesting if the user gets the case of the sub item wrong e.g. they type 'mark' instead of 'Mark'.

It's not very friendly in terms of user experience to give the user an error message if they do that so I suggested that we just make the look up of the sub item case insensitive

[source,text]
----

items/234/subItem/mark
----

would therefore find us the 'Mark' sub item.

Duncan pointed out that we'd now have more than 1 URI for the same document which isn't particularly great since theoretically there should be a one to one mapping between a URI and a given document.

He pointed out that we could do a look up to find the 'canonical identifier' before we did the redirect such that if you typed in 'mark':

[source,text]
----

items/234?subItem=mark
----

would redirect to:

[source,text]
----

items/234/subItem/Mark
----

The logic for checking the existence of a sub item would be the bit that's case insensitive and makes it more user friendly.