Check type of argument before call comparator in sort

This fixes CC#1516

>> sort/compare [1 2 #[unset!]] :>
; The R3 process crashes at this point without a system exception
[Linux] Print backtrace in case of crashing

the executable needs to be compiled with -g -rdynamic
-fvisibility=default
[Makefile] change visibility to default

Such that backtrace can extract the function symbols