More efficient way to check & insert listener into ptr_map.

Change-Id: I09ec8ed8f05621b72984a34f47db50d20ebd6c06