Also remove old signatures when rebuilding