Updated DebugBackend::UpdateHookMode to handle return hook events correctly

Avoid allocating and freeing a string buffer every Lua hook callback by keeping a std::string around in the vmstate and reusing it
Added a lock around switching the mode of Lua hook in various places