Rework SvtListener and SvtBroadcaster internals.

The old code was simply awkward.

Change-Id: I1a58a9af86c100be238d306570b40f70c5100314
No need to individually remove listeners when being destroyed.

Otherwise a crash would ensue.

Change-Id: I9a52524cc205765d059745e9f7e914b636667cb6