Use a dict for clients memoization

This is in preperation for the client being contributed by a plugin
rather than inline.

Change-Id: I191f848ccb97e1cc975ea702016b5637eab0c976
Merge "Use a dict for clients memoization"