Do not call magit-refresh-buffer for non-magit-mode buffer

fixes #538