It is not possible to remove a broker directly from a mutual fund. The only possible way is to switch from a Direct Mutual Fund Plan to Regular Mutual Fund Plan.
However, if you don’t want to switch, then you can only move from Mutual Fund A to Mutual Fund B upon request. But in this case, you have to incur the exit load and capital gain tax. So, it is better to consult your financial planner before taking such a step.