Well, start gpg-agent with a DISPLAY please.