If there is already a nonelectrified wire on the fence, bring that wire on to the ground terminal. If there are steel culverts under a road along the fence, you have a great ground. Drill it and run a wire to it. It still had hundreds of volts showing on the voltmeter. If you, your neighbors, and your power company are not concerned with stray voltage, you will likely have fewer lightning problems by bonding the charger ground to the utility ground.
Lightning often hits the power line and has to go through the charger to get to the charger ground.
A surge suppressor between the two ground fields would probably be the best way to go.