Only allow peers from which messages have been received lately to be added to swift for each endpoint