> set def=xxx
> gvim $def
> we can no longer edit the $def file; gvim tries to edit xxx and there is no way to escape the $ that I can find.

If this is in the Windows shell (cmd.exe, powershell too I believe), then `^` is the escape character. Did you try `gvim ^$def` ?