David Cameron , the Conservative leader , said : " The whole point of the Speaker is that he must be above party politics , so it wouldn 't be right for the official opposition to call for him to go or to force him to go , and I won 't be doing that .