One part of the body affected by anxiety and stress is the nervous system, which plays a primary role in basic functions like memory and learning. As a result, persistent anxiety and memory loss are associated.