Yeah, this same problem comes up in lots of places (mostly with numpy) and it's not handled very gracefully anywhere.  I think I'll leave this like it is, and maybe open a new issue for fixing optional imports across the board.