improve shadow password handling on SmartOS

Fixes #9874
Also fixes an unreported error that when half of this fix is applied
password setting succeeds but appears to fail due to lack of reading
password hashes from the shadow file.