unix: don't abort if mutex init fails

Instead of SIGABRT when mutex init fails gracefully return an error code
and cleanup.

Related to pull request #1522 
return -... instead.