The phases are always computed with a vector average because of the circular nature of phase. The scalar averaged spectrum contains contributions from emission in the whole primary beam. If the source is resolved, then different baselines will have different amplitudes, and the scalar average will relect that. Note that channels which contain only noise will appear biased after the scalar average at the mean noise level of the individual visibilities. Any interference will be obvious in this plot.
Any interference generally has semi-random phase so it tends to average out.
Alternatively you could restrict the average to the short baselines so that the exact values of offset are less critical. This option divides the frequency dependent visibilities by the channel 0 visibilities. The remaining response across the band is because of the filters and electronics. Experiment with this option, and see how much it affects the outcome. Make sure there is no severely corrupted data in the channel 0 data channels if you use this option.