gpg-agent does need HOME set

Without HOME set gpg-agent doesn't seem to want to forward a pinentry
over ssh. It otherwise works. Strange...