Make disabling pdf-cache-prefetch-minor-mode safer.

* lisp/pdf-cache.el (pdf-cache-prefetch-minor-mode): Assert PDF buffer
  only when enabling.