Fix recursive math macro crash (bug #9140)

This is different from bug #8999, since in this case a new macro instance is
created. You still get a TeX capacity exceeded error if you try to typeset the
exported document, but this is the same as for bug #8999 and better than a
crash.