My own guess would be that it is a religious reference: that the three wise men referred to are, the Father, Son and the Holy Ghost.

You have to go to Don McLean's 'American Pie' for this.
 
"...and the three men I admire most, the Father, Son, and the Holy Ghost, they caught the last train for the coast, the day the music died, and they were singing..." 

The wise men come down from heaven (Blunt). Having caught the last train for the coast (McLean), those three men buy a semi by the sea (Blunt). 

On this basis, this song may be a general rant against human nature, mankind having mainly turned their backs on God. 

On earth, the 'wise men' (Father, Son and Holy Ghost) can't escape from bad human beings, and are actually sorry to have created mankind in the first place (mankind generally being "pernickety little bastards in your fancy dress / Who just judge each other and try to impress".) 

God only made man because he was "trying to have some fun", but he made a mess of the product (i.e. human nature is imperfect) and now he can't 'escape from' mankind, can't 'be free from' mankind, And has 'no way out' of the problem that mankind is, and the imperfect creation God has made on earth. So God takes refuge from us on the coast.
 
As individuals, and as the human race collectively, if we proceed in a godless way, we have to ask ourselves the question "Where are we now?" 

Blunt isn't alone however, as he himself has God. 

Only my guess. Pure speculation. Probably well wide of the mark, I suppose someone else will tell me.