improve/fix windows backtraces and win64 memory over-consumption issue

cherry-pick of 'Merge pull request #9190 from JuliaLang/jn/win32_backtraces'