Only delete files that have a matching group + prefix.

Fixes #3873