Don't attempt to show group icons for former members in PMs

Signed-off-by: Michael Eshom <oldiesmann@oldiesmann.us>