Don't warn about unsafe functions which don't need to be unsafe

Closes #5954