There are only a few ways that a deeded easement can be eliminated as follows:1. by abandonment of the dominant owner;2. express conveyance by the dominant owner to the servient owner;3. adverse possession.Your neighbor can sue you over the easement issue but the chances of success are remote from what you have written. I suggest that you consult with a real estate attorney further about your matter to try and diffuse the situation.