Release python/swig ownership when adding object changes scope

This should fix #462