Fix a potential problem with removing old SSRCs.