Don't catch ImportError unless it is related to the non-package import

Fixing #38 prevented legitimate import errors from being propogated.