Matches keeps an internal reference to spec object

Fixes #1729
Let's open a separate PR for that.