Clean up handling of per_descriptor destruction