Reduce heap allocations on initialization
The shared_ptr to our signal that is used by connections is now deferred

No heap allocations will occur until a connection is made.
Fixed swap for m_self