Remove common unexisted prefixes too