Check type of argument before call comparator in sort

This fixes CC#1516

>> sort/compare [1 2 #[unset!]] :>
; The R3 process crashes at this point without a system exception
[Linux] Print backtrace in case of crashing

the executable needs to be compiled with -g -rdynamic
-fvisibility=default
[Makefile] change visibility to default

Such that backtrace can extract the function symbols
Check overflow in Loop_Integer

or this code will cause an infinite loop:

>> a: 9223372036854775807
== 9223372036854775807

>> for b 0 a to integer! a / 2 [print b]

With this commit:
>> for b 0 a to integer! a / 2 [print b]
0
4611686018427387904
** Math error: math or number overflow
** Where: for
** Near: for b 0 a to integer! a / 2 [print b]
Avoid negate MIN_I64

As -MIN_I64 > MAX_I64, it's undefined. Avoid this by put it in the
special case.

Reported by GCC with -fsanitize=undefined.