If the right of way is on the deed or title (it is an actual easement), then you cannot "take it back," even if you are having a dispute with the person usng the right of way: an easement is essentially part of your land and cannot be changed by the owner, any more than you could simply change your property boundaries. If it was a non-easement contract, then bring the contract to an attorney to review with you: there is no way to answer this question in the abstract, since every contract is governed by its specific langauge or terms.