He has learned that the most comfortable place to be is back there, out of your personal space. Plan what you are going to do. For example, spot a weed or a piece of manure and plan to stop him right there. Then stop him right there not 10 feet past it, or even one foot past it. Work at being definite, but don't overload your horse or yourself.
Many humans are assertive by nature, wanting to pull, poke, and pry.
Others are prone to suggesting, and must learn to be more definite about asking. A horse that is asked has a more willing attitude than one that is forced. Pay attention to the horse's feet. Choose one foot, and start noticing every time he picks up that foot. Then notice every time he is putting it down.