Don't attempt to show group icons for former members in PMs

Fixes #530

Signed-off-by: Michael Eshom <oldiesmann@oldiesmann.us>