iParadigms/walker — pull request #117 by pchristopher1275 (2014-12-18 20:16:28+00:00)
Title: Trn 333

Original ticket:

We want a simple CLI utility for any walker operator to call to get info about a link, and show content body if they have enabled walker for crawling page contents. This will be useful for diagnostics and in tutorials Something like:
walker readlink -u http://golang.org # display request results and content on stdout
It should also:
return a non-zero error code and useful message on stderr if we do not have that link at all
return information regarding only the very last fetch, or about the link itself if we have never fetched it
default to showing metadata and body (if the user is storing HTTP bodies in cassandra), but have flags to get just the body content or just the HTTP headers and metadata
It may be useful to develop this in concert with TRN-126

--end--

One thing to note right at the outset is that this patch has pre-merged TRN-333!! This ticket (implicitly) made reference to that ticket, so I merged 333 at the outset. ***If you haven't looked at TRN-333, it might be best to start with that ticket.***

I feel like this patch will be unusually controversial, given the scope of the ticket. The issue isn't really the production code, which is straight forward. But rather the testing code is the issue. Several challenges manifested during test writing.

(a) First, tests required that I build some structure to capture the output of the new cobra command. Would have liked to just pipe stdout/stderr around, but that turned out to clash with the test framework, and would have been insufficient anyway, given that exit (for which there exists no easy trap) was also an issue.

(b) My addition of FindLink to MockDatastore caused a cyclic dependency between helpers and cassandra namespace. I just moved LinkInfo into the walker namespace to resolve. One could argue that LinkInfo is a part of the interface (which is why I put it in interface.go). But, I don't know if this is the right place to put it, or if there is a preferable way to rewire mocks so that we can drop the dependency.

At this point, in reference to (b), I would like to revisit the namespacing we have in the project. Though do I like the logical distinctions put forth by the helpers, cassandra, walker, and console namespaces; I don't know if they provide any real practical benefit (***am I missing something with this?***). But, at the same time, they will from time-to-time trigger this kind of cyclic-dependency problem. And those sorts of problems require a fair bit of effort to juggle the symbols into the right final resting place: not to mention that we will sometimes have to swallow a construction that might not be what we want logically, but is needed to ease a cycle problem. Perhaps we can visit the idea of flattening the project to a single walker namespace? Or slightly more involved: use a public facing walker namespace, and a private walker namespace (or internal namespace if we have the right version of go)? And with either of these directions, figure some other way to create the logical distinctions we want. Food for thought.