compilation failure with BUILD/compile-amd64-valgrind-max
MDEV-5905: Creating tmp. memory table kills the server

The reason was that a couple of variables that hold number of rows that was used to calculate buffers was uint and caused an overflow.

Fixed by changing variables that could hold number of rows from uint to ulong and also added a cast for this test.
MDEV-5955 Server crashes in handler::ha_external_lock or assertion `m_lock_type == 2' fails in handler::ha_close on disconnect with a locked temporary table

first unlock locked tables, then close and remove temporary