After a difficult summer which has seen the Tories fall behind Labour in the polls , Mr Cameron brushed aside criticism of his own leadership by senior figures such as former party chairman Lord Saatchi and former treasurer Lord Kalms .