With the Tories rampant after the prime minister 's decision not to call an autumn election , friends of Tony Blair disclosed he was " unhappy " with the direction in which Brown was taking the party .