Repository: iterative/dvc

Issue:
gdrive: looking for user credentials when configured with service account
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug. 

Example: `repro: doesn't detect input changes`
-->
dvc push looking for user credentials when configured with service account
## Description

<!--
A clear and concise description of what the bug is.
-->
I have set up dvc with gdrive, created a google cloud project, and then created a service account according to: https://dvc.org/doc/user-guide/setup-google-drive-remote. I then `dvc add` a file and `dvc push`, and receive an error indicating that my gdrive user credentials file is missing.

here is the trace when running `dvc push -v`:
```
2021-03-04 18:47:25,775 DEBUG: Check for update is enabled.
2021-03-04 18:47:26,472 DEBUG: Preparing to upload data to 'gdrive://1qOaTb9rdhBlUdHeilikWMzAu7PI_KH7m'
2021-03-04 18:47:26,472 DEBUG: Preparing to collect status from gdrive://1qOaTb9rdhBlUdHeilikWMzAu7PI_KH7m
2021-03-04 18:47:26,472 DEBUG: Collecting information from local cache...
2021-03-04 18:47:26,475 DEBUG: Collecting information from remote cache...                                         
2021-03-04 18:47:26,475 DEBUG: Querying 1 hashes via object_exists
2021-03-04 18:47:28,678 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_file': '/home/jaheyer/chipbrain/chipbrain-transformers/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'save_credentials_backend': 'json', 'service_config': {'client_user_email': 'dvc-remote-storage@dvc-remote-storage-305303.iam.gserviceaccount.com', 'client_json_file_path': '/home/jaheyer/chipbrain/chipbrain-transformers/.dvc/dvc-remote-storage-305303-2331a8a33e40.json'}}'.
2021-03-04 18:47:28,680 ERROR: failed to push data to the cloud - Failed to authenticate GDrive remote: [Errno 2] No such file or directory: '/home/jaheyer/chipbrain/chipbrain-transformers/.dvc/tmp/gdrive-user-credentials.json'
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jaheyer/miniconda3/envs/transformers/lib/python3.7/site-packages/dvc/fs/gdrive.py", line 258, in _drive
    gauth.ServiceAuth()
  File "/home/jaheyer/miniconda3/envs/transformers/lib/python3.7/site-packages/pydrive2/auth.py", line 98, in _decorated
    self.LoadCredentials()
  File "/home/jaheyer/miniconda3/envs/transformers/lib/python3.7/site-packages/pydrive2/auth.py", line 329, in LoadCredentials
    self.LoadCredentialsFileJson()
  File "/home/jaheyer/miniconda3/envs/transformers/lib/python3.7/site-packages/pydrive2/auth.py", line 366, in LoadCredentialsFileJson
    filename=credentials_file, scopes=scopes
  File "/home/jaheyer/miniconda3/envs/transformers/lib/python3.7/site-packages/oauth2client/service_account.py", line 219, in from_json_keyfile_name
    with open(filename, 'r') as file_obj:
FileNotFoundError: [Errno 2] No such file or directory: '/home/jaheyer/chipbrain/chipbrain-transformers/.dvc/tmp/gdrive-user-credentials.json'
```
### Reproduce
N/A - don't know how to create a MWE involving service accounts and credentials. 

### Expected

<!--
A clear and concise description of what you expect to happen.
-->
I expect DVC to push my files to the gdrive.
### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc version`:**

```console
$ dvc version
DVC version: 2.0.1 (pip)
---------------------------------
Platform: Python 3.7.10 on Linux-4.15.0-46-generic-x86_64-with-debian-buster-sid
Supports: gdrive, http, https
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sda
Caches: local
Remotes: gdrive
Workspace directory: ext4 on /dev/sda
Repo: dvc, git
```

**Additional Information (if any):**
contents of my .dvc/config:
```
[core]
    remote = storage
['remote "storage"']
    url = gdrive://1qOaTb9rdhBlUdHeilikWMzAu7PI_KH7m
    gdrive_use_service_account = true
    gdrive_service_account_json_file_path = dvc-remote-storage-305303-2331a8a33e40.json
    gdrive_service_account_user_email = dvc-remote-storage@dvc-remote-storage-305303.iam.gserviceaccount.com
```

I have used user credentials to authenticate and push to this drive folder in the past, but since then I have `rm -rf .dvc/`, uninstalled dvc 1.3 and installed dvc 2.0.1


Discussion:
Discord context https://discord.com/channels/485586884165107732/563406153334128681/817180163988652052
It can be a regression in the latest PyDrive2 PR that introduced JSON support - https://github.com/iterative/PyDrive2/issues/96 . Waiting for @isidentical response to see if I'm missing something.

Resolving patch:
diff --git a/dvc/fs/gdrive.py b/dvc/fs/gdrive.py
--- a/dvc/fs/gdrive.py
+++ b/dvc/fs/gdrive.py
@@ -3,6 +3,7 @@
 import os
 import posixpath
 import re
+import tempfile
 import threading
 from collections import defaultdict
 from contextlib import contextmanager
@@ -208,8 +209,17 @@ def _drive(self):
         from pydrive2.auth import GoogleAuth
         from pydrive2.drive import GoogleDrive
 
-        if os.getenv(GDriveFileSystem.GDRIVE_CREDENTIALS_DATA):
-            with open(self._gdrive_user_credentials_path, "w") as cred_file:
+        temporary_save_path = self._gdrive_user_credentials_path
+        is_credentials_temp = os.getenv(
+            GDriveFileSystem.GDRIVE_CREDENTIALS_DATA
+        )
+        if self._use_service_account:
+            temporary_save_path = os.path.join(
+                tempfile.gettempdir(), "google-creds.json"
+            )
+
+        if is_credentials_temp:
+            with open(temporary_save_path, "w") as cred_file:
                 cred_file.write(
                     os.getenv(GDriveFileSystem.GDRIVE_CREDENTIALS_DATA)
                 )
@@ -217,6 +227,7 @@ def _drive(self):
         auth_settings = {
             "client_config_backend": "settings",
             "save_credentials": True,
+            "save_credentials_backend": "file",
             "save_credentials_file": self._gdrive_user_credentials_path,
             "get_refresh_token": True,
             "oauth_scope": [
@@ -226,13 +237,16 @@ def _drive(self):
         }
 
         if self._use_service_account:
-            auth_settings["save_credentials_backend"] = "json"
             auth_settings["service_config"] = {
                 "client_user_email": self._service_account_user_email,
                 "client_json_file_path": self._service_account_json_file_path,
             }
+            if is_credentials_temp:
+                auth_settings["service_config"][
+                    "client_json_file_path"
+                ] = temporary_save_path
+
         else:
-            auth_settings["save_credentials_backend"] = "file"
             auth_settings["client_config"] = {
                 "client_id": self._client_id or self.DEFAULT_GDRIVE_CLIENT_ID,
                 "client_secret": self._client_secret
@@ -267,8 +281,8 @@ def _drive(self):
         except Exception as exc:
             raise GDriveAuthError(self.credentials_location) from exc
         finally:
-            if os.getenv(GDriveFileSystem.GDRIVE_CREDENTIALS_DATA):
-                os.remove(self._gdrive_user_credentials_path)
+            if is_credentials_temp:
+                os.remove(temporary_save_path)
 
         return GoogleDrive(gauth)
 
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -95,7 +95,7 @@ def run(self):
 # Extra dependencies for remote integrations
 
 gs = ["gcsfs>=0.7.2"]
-gdrive = ["pydrive2>=1.7.3", "six >= 1.13.0"]
+gdrive = ["pydrive2>=1.8.1", "six >= 1.13.0"]
 s3 = ["boto3>=1.9.201"]
 azure = ["adlfs>=0.6.3", "azure.identity>=1.4.0", "knack"]
 # https://github.com/Legrandin/pycryptodome/issues/465


Test patch: