Q:

My nemeses are rampaging the countryside.  Does it matter? 

I've got a whole lot of nemeses running looking angry and apparently blowing fire at the citizens of Norende, or some such.  

In the 4 days they've been running rampant, I haven't seen any specific negative consequences of ignoring them.  Shops are getting destroyed, my citizen count doesn't go down, and they don't seem to be blocking the path to things I want to build. 
Are they just misunderstood creatures that want to be left alone, and won't bother me if I don't bother them? 

A:

There is no consequence to leaving Nemeses running around other than the fact that new Nemeses can come in and overwrite whatever Nemesis you had there previously.  So, it's possible for that super-special L99 Nemesis you have there to be overwritten by a puny L10 Beelzebub. 
To keep this from happening, check the Nemesis you want to keep from being overwritten and select the "Protect/Unprotect" option.