You shouldn’t need to remove the fence post unless it’s rotten or broken. You just need to straighten it and reinforce the base of the post.