Use raw pointers instead of ref count
Added more asserts and cleanup
p4#: 20891