Relying on the `default-directory` is not a good idea, since many packages mess with it. I'd suggest the addition of some helper method that first checks if the current-buffer is visiting a file and fallback to the default-directory only if it's not.