RegisterHotkeys: Fix a rare crash.

Use the Windows API more correctly and add more error checking.

This commit fixes #2971.