memoize descriptor -> specification mappings. This is less for performance and more to be able to support recursion