Refactoring and various improvements (ref, locking)

Improve reference updater, update reference every tick, try harder to
avoid locking issues