Improve pointer ownership

Although the work is not complete, the previous code abused
shared_ptr usage in many places where the expected ownership didn't
require it.

A lot of changes to references and unique_ptr have been done.
Merge branch 'ptrsanity'