Remove the prefix file when no more packages have a prefix
Apply patches and substitute files before removing a package.

Fix #518