joyent/smartos-live — pull request #351 by tylerflint (2014-09-19 17:49:39+00:00)
Title: metadata process crashes when smartos zone is constrained by disk space

I ran into an interesting scenario where a metadata socket was created inside of a zone, and then over time that zone filled up all of the available disk space. The socket inside of the zone failed, which then caused the metadata process on the global zone to crash. At that point, all zones are unable to communicate of the mdata-get socket, and the metadata service on the global zone will crash repeatedly.

This issue could be reproduced if desired simply by filling up all of the disk on a SmartOS zone and triggered by deleting the /.zonecontrol/metadata.sock

The fix was simple, rather than throwing an error (which is uncaught and thus crashes the process), we can simply catch the specific ENOTSOCK error, close that zone specific metadata server, and start the retry timer. This allows for either 1) the zone in question to free disk space until the socket can be created again successfully or 2) the zone to be shutdown which will remove the timers anyway.

Thanks