Allow Pyro to exclude empty patch transforms