Revert "Remove unnecessary code because we always pass spec_pattern in allocator"

This reverts commit d632f8362c3fa485867fe8e807ce8eec88ce2a79.