Yes, you need to be worried. Generally, an oral or verbal agreement can be enforced, but the big problem with one is proving its terms. If the neighbor believes she gave you permission to do the work as long as it was not too disruptive or was done in a reasonable period of time and now revokes any permission, it can be difficult to prove what was said or meant. Also, what you are discuss is not merely permission to work on her property; it's to create a permanent right to build (part of the wall) on her property, which is usually created by an easement, which *must* be in writing. You should consult with a real estate attorney about legally, how to enforce the rights you need and believe you have; you should also consider how, as a practical matter, to mollify your neighbor, such as by perhaps paying her for her inconvenience or modifyng the wall so it gets done quicker and is less intrusive. Your attorney can advise you on this, too.