Yes, the current owner can take back the land, and require you to remove the concrete at your expense and restore the land. The oral agreement you had with the prior owner is simply not binding on anyone who acquires the property from or after them and did not adjust the property lines. It was an agreement only between the two of you and bound no one else. It was, unfortunately, risky for you to rely on that owner's permission without getting that permission reflected in either an adjustment in the boundaries (e.g. by him quitclaiming that land to you) or an easement which was then filed. The current owner can enforce the property's boundaries, as reflected in the filed deeds.