markhneedham/blog huggingface dataset:
Learning Android: 'Unable to start service Intent not found'
Date: 2012-01-01 03:22:34

In the Android application that I've been playing around with I wrote a service which consumes the Twitter streaming API which I trigger from the app's main activity like so:

[source,java]
----

public class MyActivity extends Activity {
    ...
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);

        Intent intent = new Intent(this, TweetService.class);
        startService(intent);
        ...
    }
}
----

Where +++<cite>+++TweetService+++</cite>+++ is defined roughly like this:

[source,java]
----

public class TweetService extends IntentService {
    @Override
    protected void onHandleIntent(Intent intent) {
      // Twitter streaming API stuff goes here
    }
}
----

Unfortunately when I tried to deploy the app the service wasn't starting and I got this message in the log:

[source,text]
----

01-01 03:10:31.758: WARN/ActivityManager(106): Unable to start service Intent { cmp=com.example/.TweetService }: not found
----

What I hadn't realised is that the service needs to be specified in the +++<cite>+++AndroidManifest.xml+++</cite>+++ file but http://stackoverflow.com/questions/6313793/unable-to-start-service-intent-cmp-com-marie-mainactivity-backgroundservice[not inside the activity definition]:

[source,xml]
----

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android" package="com.example" android:versionCode="1" android:versionName="1.0">
    <application android:label="@string/app_name">
        <activity android:name="MyActivity" android:label="@string/app_name" android:launchMode="singleInstance">
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>
                <category android:name="android.intent.category.LAUNCHER"/>
            </intent-filter>
        </activity>
        <service android:name="TweetService"></service>
    </application>
    <uses-permission android:name="android.permission.INTERNET"/>
</manifest>
----

After adding the service definition it works fine.
Developer machine automation: Dependencies
Date: 2012-01-24 23:16:52

As I http://www.markhneedham.com/blog/2012/01/18/installing-puppet-on-oracle-linux/[mentioned in a post last week] we've been automating the setup of our developer machines with puppet over the last week and one thing that we've learnt is that you need to be careful about how you define dependencies.

The aim is to get your scripts to the point where the outcome is reasonably deterministic so that we can have confidence they're going to work the next we run them.

We noticed two ways in which we haven't quite achieved determinism yet:

== Accidental Dependencies

The first few times that we ran the scripts on top of a vanilla image we were doing it on a virtual machine which had VMware tools installed on it.

We'd forgotten that VMware tools had been installed on those VMs and ran into a problem with Oracle dependencies not being satisfied when we ran puppet on some machines which had CentOS installed directly (i.e. not on a virtual machine).

Those dependencies had been satisfied by our VMware tools installation on the VMs so we didn't realise that we hadn't explicitly stated those dependencies, something which we have done now.

== External Dependencies

We couldn't find the Firefox version that we wanted install on the default yum repositories so we created a puppet task which linked to a Firefox RPM on an external server and then installed it.

It worked originally but at some stage over the last couple of weeks the URI was changed as a minor version had been upgraded, breaking our script.

We also came across another way that external dependencies can fail today - if a corporate proxy blocks access to the URL!

We're trying to get to the stage where we're only relying on artifacts either coming from a *yum repository or an internal repository* where we can store any libraries which aren't available through yum.

== Don't assume determinism

While trying to solve these dependency problems in our puppet scripts I made the mistake of assuming that if the script runs through once and works that it's always going to be that way in the future.

Since we had achieved that previously in my mind it was impossible for it to fail in future which stopped me from properly investigating why it had stopped working.
Oracle: exp -  EXP-00008: ORACLE error 904 encountered/ORA-00904: "POLTYP": invalid identifier
Date: 2012-01-13 21:46:58

I spent a bit of time this afternoon trying to export an Oracle test database so that we could use it locally using the +++<cite>+++http://www.orafaq.com/wiki/Import_Export_FAQ#How_does_one_use_the_import.2Fexport_utilities.3F[exp]+++</cite>+++ tool.

I had to connect to exp like this:

[source,text]
----

exp user/password@remote_address
----

And then filled in the other parameters interactively.

Unfortunately when I tried to actually export the specified tables I got the following error message:

[source,text]
----

EXP-00008: ORACLE error 904 encountered
ORA-00904: "POLTYP": invalid identifier
EXP-00000: Export terminated unsuccessfully
----

I eventually came across http://oisene.blogspot.com/2010/06/error-when-using-11g-export-client-on.html[Oyvind Isene's blog post which pointed out that you'd get this problem if you tried to export a 10g database using an 11g client] which is exactly what I was trying to do!

He explains it like so:

____
The export command runs a query against a table called EXU9RLS in the SYS schema. On 11g this table was expanded with the column POLTYP and the export command (exp) expects to find this column.
____

I needed to download the 10g client so that I could use that version of exp instead. I haven't quite got it working yet but at least it's a different error to deal with!
Wireshark: Following HTTP requests/responses
Date: 2012-01-14 23:20:44

I like using http://www.wireshark.org/[Wireshark] to have a look at the traffic going across different interfaces but because it shows what's happening across the wire by the packet it's quite difficult to tell what a request/response looked like.

I've been playing around with https://github.com/caelum/restfulie-java[restfulie]/http://vraptor.caelum.com.br/[Vraptor] today so I wanted to be able to see the request/response pair when something wasn't working.

I didn't know it was actually possible but http://stackoverflow.com/questions/2163636/mapping-http-requests-to-http-responses[this post on StackOverflow describes how].

First we need to select the row which contains any part of our request/response - in this case I just selected the row representing the request - and then we go to the Analyze menu and click 'Follow TCP Stream':

image::{{<siteurl>}}/uploads/2012/01/follow_tcp_stream.jpg[Follow tcp stream,447]

We can then see the requests/responses which happened all next to each other:

image::{{<siteurl>}}/uploads/2012/01/show_stream.jpg[Show stream,500]

The keyboard shortcut to get to that menu is 'Alt-A F' but for some reason the 'Alt' key wasn't working for me by default so I had to follow the http://francisnorth.blogspot.com/2009/07/how-to-get-alt-key-to-function-properly.html[instructions on Francis North's blog] to get it working.
Haskell: Building a range of numbers from command line arguments
Date: 2012-06-03 20:13:54

I'm working through some of the http://mitpress.mit.edu/sicp/[SICP problems] in Haskell and for http://www.billthelizard.com/2010/02/sicp-exercise-122-timed-prime-test.html[problem 1.22] you need to write a function which will indicate the first 3 prime numbers above a starting value.

It is also suggested to only consider odd numbers so to find the prime numbers above 1000 the function call would look like this:

[source,haskell]
----

> searchForPrimes [1001,1003..]
[1009,1013,1019]
----

I wanted to be able to feed in the range of numbers from the command line so that I'd be able to call the function with different values and see how long it took to work it out.

I initially tried passing the above array to the program as a command line argument wrapped in a string and then parse it using +++<cite>+++read+++</cite>+++ which works if you provide a complete array:

[source,haskell]
----

> read "[1,2]" :: [Int]
[1,2]
----

But not so well for ranges:

[source,haskell]
----

> read "[1,2..]" :: [Int]
*** Exception: Prelude.read: no parse
----

Instead I thought I could just provide the first two values for the proposed range and then work from there.

To help me do this I came across the +++<cite>+++enumFromThen+++</cite>+++ function which has the following signature:

[source,haskell]
----

> :t enumFromThen
enumFromThen :: Enum a => a -> a -> [a]
----

From the Haskell source this function seems to call the literal range syntax which I mentioned above:

[source,haskell]
----

enumFromThen x y = map toEnum [fromEnum x, fromEnum y ..]
----

I was actually expecting there to be a literal range syntax definition which made a call to the various enumFrom... methods since the literal syntax seems more like http://en.wikibooks.org/wiki/Haskell/Syntactic_sugar[syntactic sugar] on top of the function call approach but apparently not!

I use it like this:

[source,haskell]
----

main = do
  args <- getArgs
  let (first:second:_) = map read args in
    putStrLn $ show $ searchForPrimes (enumFromThen first second)
----

Which I suppose could just as easily be written like this:

[source,haskell]
----

main = do
  args <- getArgs
  let (first:second:_) = map read args in
    putStrLn $ show $ searchForPrimes ([first, second..])
----

And I've gone full circle!
CSV parsing/UTF-8 encoding
Date: 2012-06-10 23:30:23

I was recently trying to parse a CSV file which I'd converted from an Excel spreadsheet but was having problems with characters beyond the standard character set.

This is http://stackoverflow.com/questions/1549139/ruby-cannot-parse-excel-file-exported-as-csv-in-os-x[an example] of what was going wrong:

[source,ruby]
----

> require 'csv'
> people = CSV.open("sponsors.csv", 'r', ?,, ?\r).to_a
["Erik D\366rnenburg", "N/A"]

> people.each { |sponsee, sponsor| puts "#{sponsee} #{sponsor}" }
Erik D?rnenburg N/A
----

I came across a Ruby gem called +++<cite>+++http://snippets.aktagon.com/snippets/159-Detecting-file-data-encoding-with-Ruby-and-the-chardet-RubyGem[chardet]+++</cite>+++ which allowed me to work out the character set of Erik's name like so:

[source,ruby]
----

> require 'chardet'
> require 'UniversalDetector'

> UniversalDetector::chardet("Erik D\366rnenburg")
=> {"encoding"=>"ISO-8859-2", "confidence"=>0.879630020576305}
----

I'd forgotten that you can work out the same thing by making use of +++<cite>+++file+++</cite>+++ like so:

[source,text]
----

> file sponsors.csv
sponsors.csv: ISO-8859 text, with CR line terminators
----

We can then make use of +++<cite>+++http://docs.moodle.org/22/en/Converting_files_to_UTF-8[iconv]+++</cite>+++ to change the file encoding like this:

[source,text]
----

> iconv -f iso-8859-2 -t utf-8 sponsors.csv > sponsors_conv.csv
----

[source,text]
----

> file sponsors_conv.csv
sponsors_conv.csv: UTF-8 Unicode text, with CR line terminators
----

Now if we parse the UTF-8 encoded file it doesn't ruin Erik's name!

[source,ruby]
----

> people = CSV.open("sponsors.csv", 'r', ?,, ?\r).to_a
["Erik D\303\266rnenburg", "N/A"]

> people.each { |sponsee, sponsor| puts "#{sponsee} #{sponsor}" }
Erik Dörnenburg N/A
----

Hopefully I'll now remember what to do next time I come across this problem!
Haskell: Mixed type lists
Date: 2012-06-19 23:09:39

I've been continuing to work through the exercises in http://www.amazon.co.uk/The-Little-Schemer-Daniel-Friedman/dp/0262560992/ref=sr_1_1?ie=UTF8&qid=1340144213&sr=8-1[The Little Schemer] and came across a problem which needed me to write a function to take  a mixed list of Integers and Strings and filter out the Integers.

As I http://www.markhneedham.com/blog/2012/06/19/the-little-schemer-attempt-2/[mentioned in my previous post] I've been doing the exercises in Haskell but I thought I might struggle with that approach here because Haskell collections are homogeneous i.e. all the elements need to be of the same type.

I read about http://en.wikibooks.org/wiki/Haskell/Existentially_quantified_types#Example%3a_heterogeneous_lists[existentially quantified types] but they seemed a bit complicated and instead I decided to use the http://www.haskell.org/ghc/docs/latest/html/libraries/base/Data-Dynamic.html[Dynamic] interface.

Using Dynamic we can define a function to strip out the numbers like this:

[source,haskell]
----

import Data.Dynamic
import Data.Maybe

noNums :: [Dynamic] -> [Dynamic]
noNums lat = cond [(null lat, []),
                   (isNumber (head lat), noNums (tail lat)),
                   (otherwise, head lat : noNums (tail lat))]

justInt :: Dynamic -> Maybe Int
justInt dyn = fromDynamic dyn :: Maybe Int

isNumber :: Dynamic -> Bool
isNumber x = isJust $ justInt x
----

We can then call the function like this:

[source,haskell]
----

> map toString $ noNums [toDyn (5 :: Int), toDyn "pears", toDyn (6 :: Int), toDyn "prunes", toDyn (9 :: Int), toDyn "dates"]
[Just "pears",Just "prunes",Just "dates"]
----

[source,haskell]
----

toString :: Dynamic -> Maybe String
toString dyn = fromDynamic dyn
----

+++<cite>+++fromDynamic+++</cite>+++ eventually makes a call to +++<cite>+++http://www.haskell.org/ghc/docs/7.2.2/html/libraries/ghc-prim-0.2.0.0/GHC-Prim.html#v:unsafeCoerce-35-[unSafeCoerce#]+++</cite>+++:

____
The function unsafeCoerce# allows you to side-step the typechecker entirely. That is, it allows you to coerce any type into any other type. If you use this function, you had better get it right, otherwise segmentation faults await. It is generally used when you want to write a program that you know is well-typed, but where Haskell's type system is not expressive enough to prove that it is well typed.
____

I wanted to try and make the 'isNumber' function handle any numeric type rather than just Ints but I haven't quite worked out how to do that.

Obviously I'm only using Dynamic here because the exercise requires it but I'm not sure what real life situation would require its use.

If anyone has used it before or knows a use case I'd be interested to know what it is!
Powerpoint saving movies as images
Date: 2012-06-30 10:05:04

I've been working on a presentation for the ThoughtWorks Europe away day over the last few days and I created some screen casts using http://www.techsmith.com/camtasia.html[Camtasia] which I wanted to include.

It's reasonably easy to insert movies into Powerpoint but I was finding that when I saved the file and then reloaded it the movies had been converted into images which wasn't what I wanted at all!

Eventually I came across http://www.ehow.com/how_4500346_keep-embedded-video-powerpoint-after.html[a blog post which explained that I'd been saving the file as the wrong format].

I'd been saving it as a 'ppt' but what I actually needed was to save it as a 'ppsx':

image::{{<siteurl>}}/uploads/2012/06/graph.png[Graph,600]

I spent about an hour trying to work out what I was doing wrong, an hour I hopefully won't have to spend next time.
Creating a Samba share between Ubuntu and Mac OS X
Date: 2012-06-24 00:40:35

On the project I'm currently working on we have our development environment setup on a bare bones Ubuntu instance which we run via VmWare.

We wanted to be able to edit files on the VM from the host O/S so my colleague http://twitter.com/philandstuff[Phil] suggested that we set up a Samba server on the VM and then connect to it from the Mac.

We first needed to install a couple of packages on the VM:

* apt-get install samba
* apt-get install libpam-smbpass

The first package is self explanatory and the second allows us to keep the samba username/password in sync with the unix user on the VM.

Installing the samba package will automatically start up the Samba daemon 'smbd'.

[source,text]
----

$ ps aux | grep smbd
mneedham 10915  0.0  0.0   7624   928 pts/14   S+   17:37   0:00 grep --color=auto smbd
root     32610  0.0  0.1  95372  5408 ?        S    Jun22   0:50 smbd -F
----

We then need to edit +++<cite>+++/etc/samba/smb.conf+++</cite>+++:

First we uncomment this line:

[source,text]
----

security = user
----

Then add a share, probably at the bottom of the file but anywhere is fine:

[source,text]
----

[mneedham]
comment = Mark's vm
read only = no
path = /home/mneedham
guest ok = no
browseable = yes
create mask = 0644
----

From the Mac we need to mount the share:

* Go to finder
* Connect to server (Cmd - K)
* Type in 'smb://ip.of-vm
* Select the name of the share

The share should now be accessible from the host O/S at +++<cite>+++/Volumes/name.of.share+++</cite>+++

Looking back I'm sure there's a way to configure VmWare to share files from the guest O/S but at least I now know another way to do it as well!
neo4j: Handling optional relationships
Date: 2012-06-24 23:32:17

On my ThoughtWorks neo4j there are now two different types of relationships between people nodes - they can either be colleagues or one can be the http://www.markhneedham.com/blog/2012/06/21/visualising-a-neo4j-graph-using-gephi/[sponsor of the other].

The graph looks like this:

image::{{<siteurl>}}/uploads/2012/06/sponsors-colleagues.png[Sponsors colleagues,392]

I wanted to get a list of all the sponsor pairs but also have some indicator of whether the two people have worked together.

I started off by getting all of the sponsor pairs:

[source,text]
----

START n = node(*)
MATCH n-[r:sponsor_of]->n2
RETURN n.name, n2.name
----

I managed to narrow that down to the people who sponsored someone that they'd worked with like so:

[source,text]
----

START n = node(*)
MATCH n-[r:sponsor_of]->n2, n-[r2:colleagues]->c
WHERE c = n2
RETURN n.name, n2.name
----

But it wasn't quite what I wanted since I'd now lost all the sponsor pairs who didn't work together.

My next attempt was to remove the WHERE clause and try the following which isn't even a valid cypher query:

[source,text]
----

START n = node(*)
MATCH n-[r:sponsor_of]->n2, n-[r2:colleagues]->c
RETURN n.name, n2.name, n2 IN [c]
----

I was struggling so I decided to draw out the above diagram and then work backwards from the type of output which I expected if I had the correct query.

The output I wanted was like this:

[source,text]
----

PersonA | PersonB | Sponsor Relationship | Colleague Relationship
PersonA | PersonC | Sponsor Relationship | -
----

Once I had written it out on paper it became clear that what I needed to do was find all the sponsor pairs and then optionally look for a colleagues relationship between the pair:

[source,text]
----

START n = node(*)
MATCH n-[r:sponsor_of]->n2-[r2?:colleagues]->n
RETURN n.name, n2.name, r, r2
----

The '?' before the ':' in the colleagues relationship indicates that it's optional and will still return the traversal even if that relationship doesn't exist.

If we run that query in the console it does exactly what we want:

[source,text]
----

==> +--------------------------------------------------------------------------------------+
==> | n.name            | n2.name            | r                      | r2                 |
==> +--------------------------------------------------------------------------------------+
==> | "PersonA"         | "PersonB"          | :sponsor_of[261255] {} | :colleagues[217292]|
==> | "PersonA"         | "PersonC"          | :sponsor_of[261252] {} | <null>             |
==> +--------------------------------------------------------------------------------------+
----
Why you shouldn't use name as a key a.k.a. I am an idiot
Date: 2012-06-24 22:55:39

I think one of the first things that I learnt about dealing with users in a data store is that you should never use name as a primary key because their might be two people with the same name.

Despite knowing that I foolishly chose to ignore this knowledge when building my neo4j graph and used name as the key for the Lucene index.

I thought I'd got away with it but NO!

Earlier today I was trying to work out who the most connected person at ThoughtWorks is and the graph was suggesting that 'Rahul Singh' was the most connected, having worked with 540 people.

I mentioned this to https://twitter.com/#!/jennifersmithco[Jen] who felt something was probably wrong since he'd only started working at ThoughtWorks a couple of years ago.

Amusingly Jen found an email from 18 months ago sent by Rahul #1 explaining that there were in fact two people with exactly the same name and he was getting emails intended for the other one and vice versa.

I now have first hand knowledge of what can happen if you ignore one of the most basic rules of software development!

My gamble that there probably wouldn't be two people with the same name in such a small dataset has totally failed and from now on I'll be sure to use a unique key!
Brightbox Repository: GPG error: The following signatures couldn't be verified because the public key is not available
Date: 2012-06-24 00:58:43

We're using the https://launchpad.net/~brightbox/+archive/ruby-ng[Brightbox Ruby repository] to get the versions of Ruby which we install on our machines and although we eventually put the configuration for this repository into Puppet we initially tested it out on a local VM.

To start with you need to add the repository to +++<cite>+++/etc/apt/sources.list+++</cite>+++:

[source,text]
----

deb http://ppa.launchpad.net/brightbox/ruby-ng/ubuntu lucid main
----

To get that picked up we run the following:

[source,text]
----

apt-get update
----

Which initially threw this error because it's a gpg signed repository and we hadn't added the key:

____
W: GPG error: http://ppa.launchpad.net lucid Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F5DA5F09C3173AA6
____

I recently realised that the instructions explaining how to sign the repository are hidden away in an overlay on the page but the http://docs.opsview.com/doku.php?id=opsview-community:repository-key[opsview wiki] also explains what to do.

To add the key we need to run the following:

[source,text]
----

sudo apt-key add -

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: SKS 1.0.10

mI0ETKTCMQEEAMX3ttL4YFO5AQ7Z6L5gaGw57CJBQl6jCv6lka0p8DaGNkeX0Rs9DhINa8qR
hxJCPK6ijeoNss69G/ni+sMSRViJBFWXzitEE1ew5YM2sw7wLE3guToDu60kaDwIn5mR3GTx
cgqDrQeCuGZJgz3e2lgmGYw2rAhMe78rRgkR5GFvABEBAAG0G0xhdW5jaHBhZCBQUEEgZm9y
IEJyaWdodGJveIi4BBMBAgAiBQJMpMIxAhsDBgsJCAcDAgYVCAIJCgsEFgIDAQIeAQIXgAAK
CRD12l8Jwxc6pl2BA/4p5DFEpGVvkgLj7/YLYCtYmZDw8i/drGbkWfIQiOgPWIf8QgpJXVME
1tkH8N1ssjbJlUKl/HubNBKZ6HDyQsQASFug+eI6KhSFMScDBf/oMX3zVCTTvUkgJtOWYc5d
77zJacEUGoSEx63QUJVvp/LAnqkZbt17JJL6HOou/CNicw==
=G8vE
-----END PGP PUBLIC KEY BLOCK-----
----

Then press Ctrl-D to exit the command.

The public key comes from http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0xF5DA5F09C3173AA6[here] and is referenced under the https://launchpad.net/~brightbox/+archive/ruby-ng['Signing key' section].