SymPy used to do it this way.  We had `Wild` subclasses for our various objects.  It was hard to extend this idea to all of the relevant subtype so we eventually made a function like a tree-based `raw_init` that took a normal sympy expression and a list of sympy subexpressions that should be variables, and hacked together a broken expression with the variables inside.  This was a pain because at this point you couldn't do anything like print the expressoin. 

Eventually we found it was simpler to leave the patterns as a SymPy expression and a set of wild variables.  It caused much less havoc than trying to force both pieces of information into the same data structure.

https://github.com/sympy/sympy/pull/1718