No, an unrecorded restriction (e.g. one not on the deed and not filed with the county) is not binding on future buyers. If the restriction only appears, as you indicate, in the contract of sale, then it was a contractual obligation only between the two of you, and a contract is not binding on non-parties to it; it only binds the people who were part of the contract. You were restricted; those you sell to are not.