Core/Maps: Fix Spline triggered assert

Fix an assert triggered by float to int32 cast overflowing to -1, now it replaces -\ with max int32 value.
Fix another assert triggered by Vector3 magnitude float overflow to max/infinity in StaticMapTree::isInLineOfSight(), in this case return false.

Both asserts can be reproduced by casting Mind Control to a NPC, tele to z: 1.0e+38 using client hack tools, move to allow the server to register the new position and stop Mind Control.

Fixes https://github.com/TrinityCore/TrinityCore/issues/8970 and https://github.com/TrinityCore/TrinityCore/issues/10578 , might fix https://github.com/TrinityCore/TrinityCore/issues/10355 as well.