Repository: certbot/certbot

Issue:
Plugin dns-google does not work if certbot is running in GCP CloudShell environment
I'm trying to get certificates for my tests with the help of certbot running it in the GCP CloudShell environment. 
I don't want to create ServiceAccount for this purpose and assume that the dns-google plugin will obtain info from the metadata server. But it seems like there is a bug where dns-google plugin could not get the `project id` from the metadata server if it's running in CloudShell.

## My operating system is (include version):
uname -a
Linux cs-325246420689-default 5.15.65+ #1 SMP Sat Jan 21 10:12:05 UTC 2023 x86_64 GNU/Linux

## I installed Certbot with (snap, OS package manager, pip, certbot-auto, etc):
pip

## I ran this command and it produced this output:

Command (email is redacted):

```shell
pip install certbot certbot-dns-google
mkdir "$HOME/certbot"
export $USER_EMAIL="igor_lysak@example.com"
certbot certonly \
  --config-dir "$HOME/certbot" \
  --work-dir "$HOME/certbot" \
  --logs-dir "$HOME/certbot" \
  --preferred-challenges dns \
  --noninteractive \
  --email "$USER_EMAIL" \
  --agree-tos \
  --staging \
  --dns-google \
  --domain "example.com" \
  --domain "*.example.com"
```

Output:

```text
Saving debug log to /home/igor_lysak/certbot/letsencrypt.log
Requesting a certificate for example.com and *.example.com
Error finding zone. Skipping cleanup.
Error finding zone. Skipping cleanup.
Encountered error finding managed zone: <HttpError 400 when requesting https://dns.googleapis.com/dns/v1/projects//managedZones?dnsName=example.com.&alt=json returned "Invalid resource field value in the request.". Details: "[{'message': 'Invalid resource field value in the request.', 'domain': 'global', 'reason': 'invalidParameter'}]">
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /home/igor_lysak/certbot/letsencrypt.log or re-run Certbot with -v for more details.
```

## Certbot's behavior differed from what I expected because:

After some investigation, I have found that in [dns_google.py#L95 _GoogleClient constructor](https://github.com/certbot/certbot/blob/master/certbot-dns-google/certbot_dns_google/_internal/dns_google.py#L95) method `get_project_id()` returns empty string in CloudShell environment and this leads to failure of requesting resources via GCP API. In my opinion, it should fall back to the `gcloud` default configuration.

## Here is a Certbot log showing the issue (if available):

Domain name and communication with `api.letsencrypt.org` redacted.

```logs
2023-02-15 12:17:00,052:DEBUG:certbot._internal.main:certbot version: 2.2.0
2023-02-15 12:17:00,053:DEBUG:certbot._internal.main:Location of certbot entry point: /home/igor_lysak/.local/bin/certbot
2023-02-15 12:17:00,053:DEBUG:certbot._internal.main:Arguments: ['--config-dir', '/home/igor_lysak/certbot', '--work-dir', '/home/igor_lysak/certbot', '--logs-dir', '/home/igor_lysak/certbot', '--preferred-challenges', 'dns', '--noninteractive', '--email', 'igor_lysak@example.com', '--agree-tos', '--staging', '--dns-google', '--domain', 'example.com', '--domain', '*.example.com']
2023-02-15 12:17:00,053:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-google,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-02-15 12:17:00,066:DEBUG:certbot._internal.log:Root logging level set at 30
2023-02-15 12:17:00,068:DEBUG:certbot._internal.plugins.selection:Requested authenticator dns-google and installer None
2023-02-15 12:17:00,075:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * dns-google
Description: Obtain certificates using a DNS TXT record (if you are using Google Cloud DNS for DNS).
Interfaces: Authenticator, Plugin
Entry point: dns-google = certbot_dns_google._internal.dns_google:Authenticator
Initialized: <certbot_dns_google._internal.dns_google.Authenticator object at 0x7f69f0c3a790>
Prep: True
2023-02-15 12:17:00,076:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_dns_google._internal.dns_google.Authenticator object at 0x7f69f0c3a790> and installer None
2023-02-15 12:17:00,076:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator dns-google, Installer None
...[redacted]
2023-02-15 12:17:01,371:INFO:certbot._internal.auth_handler:Performing the following challenges:
2023-02-15 12:17:01,371:INFO:certbot._internal.auth_handler:dns-01 challenge for example.com
2023-02-15 12:17:01,371:INFO:certbot._internal.auth_handler:dns-01 challenge for example.com
2023-02-15 12:17:01,381:DEBUG:google.auth._default:Checking None for explicit credentials as part of auth process...
2023-02-15 12:17:01,381:DEBUG:google.auth._default:Checking Cloud SDK credentials as part of auth process...
2023-02-15 12:17:01,382:DEBUG:google.auth._default:Cloud SDK credentials not found on disk; not using them
2023-02-15 12:17:01,383:DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254
2023-02-15 12:17:01,388:DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254/computeMetadata/v1/project/project-id
2023-02-15 12:17:01,394:DEBUG:googleapiclient.discovery:URL being requested: GET https://dns.googleapis.com/dns/v1/projects//managedZones?dnsName=example.com.&alt=json
2023-02-15 12:17:01,394:DEBUG:google_auth_httplib2:Making request: GET http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/?recursive=true
2023-02-15 12:17:01,396:DEBUG:google_auth_httplib2:Making request: GET http://169.254.169.254/computeMetadata/v1/instance/service-accounts/igor_lysak@example.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform.read-only%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fndev.clouddns.readonly%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fndev.clouddns.readwrite
2023-02-15 12:17:01,520:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot_dns_google/_internal/dns_google.py", line 281, in _find_managed_zone_id
    response = request.execute()
  File "/usr/local/lib/python3.9/dist-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://dns.googleapis.com/dns/v1/projects//managedZones?dnsName=example.com.&alt=json returned "Invalid resource field value in the request.". Details: "[{'message': 'Invalid resource field value in the request.', 'domain': 'global', 'reason': 'invalidParameter'}]">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/_internal/auth_handler.py", line 88, in handle_authorizations
    resps = self.auth.perform(achalls)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/plugins/dns_common.py", line 76, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot_dns_google/_internal/dns_google.py", line 67, in _perform
    self._get_google_client().add_txt_record(domain, validation_name, validation, self.ttl)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot_dns_google/_internal/dns_google.py", line 116, in add_txt_record
    zone_id = self._find_managed_zone_id(domain)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot_dns_google/_internal/dns_google.py", line 284, in _find_managed_zone_id
    raise errors.PluginError('Encountered error finding managed zone: {0}'
certbot.errors.PluginError: Encountered error finding managed zone: <HttpError 400 when requesting https://dns.googleapis.com/dns/v1/projects//managedZones?dnsName=example.com.&alt=json returned "Invalid resource field value in the request.". Details: "[{'message': 'Invalid resource field value in the request.', 'domain': 'global', 'reason': 'invalidParameter'}]">

2023-02-15 12:17:01,520:DEBUG:certbot._internal.error_handler:Calling registered functions
2023-02-15 12:17:01,520:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-02-15 12:17:01,527:DEBUG:google.auth._default:Checking None for explicit credentials as part of auth process...
2023-02-15 12:17:01,527:DEBUG:google.auth._default:Checking Cloud SDK credentials as part of auth process...
2023-02-15 12:17:01,527:DEBUG:google.auth._default:Cloud SDK credentials not found on disk; not using them
2023-02-15 12:17:01,528:DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254
2023-02-15 12:17:01,534:DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254/computeMetadata/v1/project/project-id
2023-02-15 12:17:01,542:DEBUG:googleapiclient.discovery:URL being requested: GET https://dns.googleapis.com/dns/v1/projects//managedZones?dnsName=example.com.&alt=json
2023-02-15 12:17:01,542:DEBUG:google_auth_httplib2:Making request: GET http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/?recursive=true
2023-02-15 12:17:01,545:DEBUG:google_auth_httplib2:Making request: GET http://169.254.169.254/computeMetadata/v1/instance/service-accounts/igor_lysak@example.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform.read-only%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fndev.clouddns.readonly%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fndev.clouddns.readwrite
2023-02-15 12:17:01,661:WARNING:certbot_dns_google._internal.dns_google:Error finding zone. Skipping cleanup.
2023-02-15 12:17:01,666:DEBUG:google.auth._default:Checking None for explicit credentials as part of auth process...
2023-02-15 12:17:01,666:DEBUG:google.auth._default:Checking Cloud SDK credentials as part of auth process...
2023-02-15 12:17:01,666:DEBUG:google.auth._default:Cloud SDK credentials not found on disk; not using them
2023-02-15 12:17:01,666:DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254
2023-02-15 12:17:01,671:DEBUG:google.auth.transport._http_client:Making request: GET http://169.254.169.254/computeMetadata/v1/project/project-id
2023-02-15 12:17:01,675:DEBUG:googleapiclient.discovery:URL being requested: GET https://dns.googleapis.com/dns/v1/projects//managedZones?dnsName=example.com.&alt=json
2023-02-15 12:17:01,676:DEBUG:google_auth_httplib2:Making request: GET http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/?recursive=true
2023-02-15 12:17:01,678:DEBUG:google_auth_httplib2:Making request: GET http://169.254.169.254/computeMetadata/v1/instance/service-accounts/igor_lysak@example.com/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform.read-only%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fndev.clouddns.readonly%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fndev.clouddns.readwrite
2023-02-15 12:17:01,793:WARNING:certbot_dns_google._internal.dns_google:Error finding zone. Skipping cleanup.
2023-02-15 12:17:01,793:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot_dns_google/_internal/dns_google.py", line 281, in _find_managed_zone_id
    response = request.execute()
  File "/usr/local/lib/python3.9/dist-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/googleapiclient/http.py", line 938, in execute
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://dns.googleapis.com/dns/v1/projects//managedZones?dnsName=example.com.&alt=json returned "Invalid resource field value in the request.". Details: "[{'message': 'Invalid resource field value in the request.', 'domain': 'global', 'reason': 'invalidParameter'}]">

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/igor_lysak/.local/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/_internal/main.py", line 1736, in main
    return config.func(config, plugins)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/_internal/main.py", line 1590, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/_internal/main.py", line 138, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/_internal/client.py", line 516, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/_internal/client.py", line 428, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/_internal/client.py", line 496, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/_internal/auth_handler.py", line 88, in handle_authorizations
    resps = self.auth.perform(achalls)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot/plugins/dns_common.py", line 76, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot_dns_google/_internal/dns_google.py", line 67, in _perform
    self._get_google_client().add_txt_record(domain, validation_name, validation, self.ttl)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot_dns_google/_internal/dns_google.py", line 116, in add_txt_record
    zone_id = self._find_managed_zone_id(domain)
  File "/home/igor_lysak/.local/lib/python3.9/site-packages/certbot_dns_google/_internal/dns_google.py", line 284, in _find_managed_zone_id
    raise errors.PluginError('Encountered error finding managed zone: {0}'
certbot.errors.PluginError: Encountered error finding managed zone: <HttpError 400 when requesting https://dns.googleapis.com/dns/v1/projects//managedZones?dnsName=example.com.&alt=json returned "Invalid resource field value in the request.". Details: "[{'message': 'Invalid resource field value in the request.', 'domain': 'global', 'reason': 'invalidParameter'}]">
2023-02-15 12:17:01,794:ERROR:certbot._internal.log:Encountered error finding managed zone: <HttpError 400 when requesting https://dns.googleapis.com/dns/v1/projects//managedZones?dnsName=example.com.&alt=json returned "Invalid resource field value in the request.". Details: "[{'message': 'Invalid resource field value in the request.', 'domain': 'global', 'reason': 'invalidParameter'}]">
```
Used `google-auth` Deprecation fixed
## Pull Request Checklist
fixes #8926 

- [x] If the change being made is to a [distributed component](https://certbot.eff.org/docs/contributing.html#code-components-and-layout), edit the `master` section of `certbot/CHANGELOG.md` to include a description of the change being made.
- [x] Add or update any documentation as needed to support the changes in this PR.
- [x] Include your name in `AUTHORS.md` if you like.



Discussion:
So the metadata server returns an empty-bodied HTTP 200 for the project-id, but then apparently you're meant to look to see if `GOOGLE_CLOUD_PROJECT` is set. 🤔

Is running Certbot in Cloud Shell useful and something future users are likely to do? Owing to its lack of permanency and persistence, it seems suitable to a throwaway invocation of Certbot, which is not a use case that we're likely to claim support for or test against.
I see your point. In case, when the primary usage of Certbot is to be run as part of some automation there is nothing to do. This is undoubtedly a throwaway invocation of Certbot in the Cloud Shell environment. Thanks for the clarification.
Tests are failing on this PR; please let us know if you need help figuring out how to see test results, or understanding what test failures mean
> Tests are failing on this PR; please let us know if you need help figuring out how to see test results, or understanding what test failures mean

Yes that would be helpful to see where this is going wrong! I did consult many articles before making these changes but, cannot figure out where it is failing!
Only one test case failed with error 

`ERROR: InvocationError for command /home/vsts/work/1/s/.tox/lint/bin/python -m pylint --reports=n --rcfile=.pylintrc acme/acme certbot/certbot certbot-ci/certbot_integration_tests certbot-apache/certbot_apache certbot-compatibility-test/certbot_compatibility_test certbot-dns-cloudflare/certbot_dns_cloudflare certbot-dns-cloudxns/certbot_dns_cloudxns certbot-dns-digitalocean/certbot_dns_digitalocean certbot-dns-dnsimple/certbot_dns_dnsimple certbot-dns-dnsmadeeasy/certbot_dns_dnsmadeeasy certbot-dns-gehirn/certbot_dns_gehirn certbot-dns-google/certbot_dns_google certbot-dns-linode/certbot_dns_linode certbot-dns-luadns/certbot_dns_luadns certbot-dns-nsone/certbot_dns_nsone certbot-dns-ovh/certbot_dns_ovh certbot-dns-rfc2136/certbot_dns_rfc2136 certbot-dns-route53/certbot_dns_route53 certbot-dns-sakuracloud/certbot_dns_sakuracloud certbot-nginx/certbot_nginx tests/lock_test.py (exited with code 2)`

@alexzorin  Please guide!
@alexzorin  Now we getting this error ` Could not find a version that satisfies the requirement cffi==1.14.6` 
@alexzorin  Did the requested changes but now more tests are failing!  :''-(
@alexzorin Still lint check is failing. Investigating the issue!
re: the pylint errors, they're basically unrelated to your changes but are caused by pylint getting bumped to a new version. I think we should probably fix these in `master` before merging this PR. I'll try come back to it soon.
> re: the pylint errors, they're basically unrelated to your changes but are caused by pylint getting bumped to a new version. I think we should probably fix these in `master` before merging this PR. I'll try come back to it soon.

Thank you! do let me know if any changes are required!
Is the PyLint errors still stuck @alexzorin 
Whenever the above pylint PR is resolved, we'll also be able to merge this PR.
Hey the pylint stuff got fixed, so I think all that is left is to resolve the resulting merge conflicts, by merging the updated `master` branch:

- Regenerate `tools/requirements.txt` (you also shouldn't get that `cffi` error anymore)
- Merging the `certbot/CHANGELOG.md` entries
@alexzorin  All updated, Please do have a look!
@alexzorin  any more changes required?

Hey, I haven't forgotten this, there's just some extra complications around dependencies that I'm looking into.
@ohemorange as I hinted at today, there are a couple of packages (`python3-googleapi` and `python3-uritemplate`) that will cause problems if our aim is packagability on Bionic and Buster. Plus one more (`python3-httplib2`) which is installable in CentOS 8 only if you enable the PowerTools repo first.

Based on reading its changelog, we need `python3-google-api-client>=1.6.1` because that's when they added support for plugging in `google-auth` instead of `oauth2client`. However, that is only available in `buster-backports` and not in Bionic at all. 

`python3-uritemplate>=3.0.0` is required by `python3-google-api-client` but is only in Bullseye and Hirsute.

With PowerTools, it's not required *so far*  to install Certbot from EPEL 8, right? Not a great feeling to add that extra friction here.

I wonder if we should defer dropping `oauth2client` until after distros catch up (2022+).

| Package                                      | Required          | Buster    | Bionic    | EPEL 8         | Oldest  | 
|----------------------------------------------|-------------------|-----------|-----------|----------------|---------| 
| python3-google-auth                          | Any?                 | 1.5.1-1   | 1.3.0-2   | 1.1.1          | 1.1.1   | 
|                                              |                   |           |           |                |         | 
| python3-pyasn1                               | >=0.1.7           | 0.4.2-3   | 0.4.2-3   | 0.3.7          | 0.3.7   | 
| python3-pyasn1-modules                       | >=0.0.5           | 0.2.1-0.2 | 0.2.1-0.2 | 0.3.7          | 0.3.7   | 
| python3-rsa                                  | >=3.1.4           | 4.0-2     | 3.4.2-1   | 3.4.2          | 3.4.2   | 
| python3-six                                  | >=1.9.0           | 1.12.0-1  | 1.11.0-2  | 1.11.0         | 1.11.0  | 
| python3-cachetools                           | >=2.0.0           | 3.1.0-2   | 2.0.0-2   | 3.1.1          | 2.0.0-2 | 
|                                              |                   |           |           |                |         | 
| python3-google-api-client/python3-googleapi | >=1.6.1           | **1.5.5-1***  | **1.5.5-1**   | 1.6.7          | **?**       | 
| python3-httplib2                             | >=0.9.2,<1dev     | 0.11.3-2  | 0.9.2     | **?** (Powertools) | **?**       | 
| python3-oauth2client                         | >=1.5.0,<5.0.0dev | 4.1.2-3   | 4.1.2-3   | 4.1.3          | 4.1.2-3 | 
| python3-uritemplate                          | >=3.0.0,<4dev     | **0.6-3**     | **0.6-1**     | 3.0.0          | **?**       | 

Ok, you might have seen the discussion in mattermost. To summarize, we're only going to constrain by EPEL 8 going forward. As for PowerTools -- I'm not familiar with the patterns around that. Is it likely that the need to install PowerTools will be gone in the future when the distros catch up, or will that dependency always only live there, and we'll have to add it if we ever want it? 
>or will that dependency always only live there, and we'll have to add it if we ever want it?

Edit: doesn't matter, google already requires powertools on epel8!

I think it's this.

PowerTools is called "CodeReady Linux Builder" in RHEL8 (https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/package_manifest/codereadylinuxbuilder-repository) and that's where they've made `python3-httplib2` available.

Previously, `python3-httplib2` was just available in EPEL7.

I couldn't really find an explanation for why the separation was introduced. Maybe since Red Hat decided to package it themselves, EPEL decided not to?
So at the moment I am working in this branch: https://github.com/certbot/certbot/commits/test-google-constraints to figure out what things should look like with the new pinning system and strategy. 

Once that is all passing, we should be able to slap it on top of this PR and be good to go.
It looks like we also need [`google-auth-httplib2`](https://github.com/googleapis/google-auth-library-python-httplib2), if want to combine `google-api-python-client`  + `google-auth`. It's not packaged in EPEL8.

I caught this very late during manual testing of `CERTBOT_OLDEST=1`, because it's only a runtime dependency in `google-api-python-client==1.6.7`. It became an explicit dependency in `1.7.0`.

If we still wanna go ahead with this change, we'd need EPEL to package `google-auth-httplib2`.
Oh no, nice catch. How was one packaged in EPEL without the other?
>How was one packaged in EPEL without the other?

My understanding ...

`google-auth` is useful independently of `google-api-python-client`, like, [you can just use it as an authentication adapter for `requests`](https://google-auth.readthedocs.io/en/master/user-guide.html#requests).

`google-api-python-client` requires `httplib2`, so if you want to use `google-auth` with it, you also need the `httplib2` adapter for `google-auth`. 

This isn't a problem for the status quo (`google-api-python-client`+`oauth2client`) because `oauth2client` doesn't need a separate adapter.

Another tidbit is that Google now recommends using https://googleapis.dev/python/dns/latest/index.html instead of the monolithic `google-api-python-client` ... but of course that's not packaged either.
Any changes required?

Hi @soham4abc,

We need to put this one on hold because making this change would create some trouble for our distro packagers. Although `oauth2client` is deprecated, it's going to keep working for the foreseeable future and we're going to stick with it for a little while longer.

Sorry! I appreciate you put a lot of work into this.
> Hi @soham4abc,
> 
> We need to put this one on hold because making this change would create some trouble for our distro packagers. Although `oauth2client` is deprecated, it's going to keep working for the foreseeable future and we're going to stick with it for a little while longer.
> 
> Sorry! I appreciate you put a lot of work into this.

No problem at all! Take your time! 
I think we may be unblocked here since EPEL 8 uses Python 3.6 which we no longer support. Does that seem right to people?
If we are still supporting Python 3.7, that includes Debian Buster. The version of https://packages.debian.org/buster-backports/python3-googleapi that we need is only available from `buster-backports`, not `buster`. 

Is that a problem?  If backports is not a default repo, we can't require that version, right?
On the other hand, being a stable distro, Buster base repos are never going to upgrade Certbot. Same with Ubuntu Bionic. So maybe we ignore them.

EPEL9:
- now has `python3-httplib2`, without PowerTools, which is good
- is missing `python3-google-api-client` - https://packages.fedoraproject.org/pkgs/google-api-python-client/python3-google-api-client/. Hopefully there's not a problem with packaging this. It is packaged for Fedora 36 so I think it should be OK?

👍 on this being unblocked, I think.
I agree with you Alex and think this is unblocked. Thanks for taking a look.

(As for Debian Buster, I think one can make an argument either way, but the last time we talked about this kind of thing in general was at https://github.com/certbot/certbot/pull/8999#discussion_r704852799 where I think we largely agreed to not worry about our packages that are not seeing regular updates. Plus, [Python 3.7 reaches its EOL next year](https://devguide.python.org/versions/) so I doubt we'll want to support them much longer anyway).
just checking the status of this: it looks like we're not worried about the package dependency issues, and that @alexzorin's last review comments were addressed -- does that seem right, and is this ready to merge?
@wgreenberg it's close - I tried merging `master` into this PR and it seems to work okay. There are weird `mypy` failures which I don't understand, since the version of `mypy` didn't get bumped in the repin. (Edit: it might be due to the bump on `types-pyopenssl`).

I do think this still needs a thorough manual test, including via instance authentication on a GCP virtual machine. I have only tested this very lightly.

The PR with merged `master` can be found at https://github.com/certbot/certbot/compare/master...alexzorin:updated_pr_8928.

If you feel like picking this up, I think it should be ready to go.
This PR seems to have a lot of rebase issues. If wanted I can issue a fresh PR with all the required changes


Resolving patch:
diff --git a/certbot-dns-google/certbot_dns_google/__init__.py b/certbot-dns-google/certbot_dns_google/__init__.py
--- a/certbot-dns-google/certbot_dns_google/__init__.py
+++ b/certbot-dns-google/certbot_dns_google/__init__.py
@@ -14,7 +14,14 @@
 ========================================  =====================================
 ``--dns-google-credentials``              Google Cloud Platform credentials_
                                           JSON file.
-                                          (Required - Optional on Google Compute Engine)
+
+                                          (Required if not using `Application Default
+                                          Credentials <https://cloud.google.com/docs/authentication/
+                                          application-default-credentials>`_.)
+``--dns-google-project``                  The ID of the Google Cloud project that the Google
+                                          Cloud DNS managed zone(s) reside in.
+
+                                          (Default: project that the Google credentials_ belong to)
 ``--dns-google-propagation-seconds``      The number of seconds to wait for DNS
                                           to propagate before asking the ACME
                                           server to verify the DNS record.
@@ -25,45 +32,37 @@
 Credentials
 -----------
 
-Use of this plugin requires Google Cloud Platform API credentials
-for an account with the following permissions:
+Use of this plugin requires Google Cloud Platform credentials with the ability to modify the Cloud
+DNS managed zone(s) for which certificates are being issued.
 
-* ``dns.changes.create``
-* ``dns.changes.get``
-* ``dns.changes.list``
-* ``dns.managedZones.get``
-* ``dns.managedZones.list``
-* ``dns.resourceRecordSets.create``
-* ``dns.resourceRecordSets.delete``
-* ``dns.resourceRecordSets.list``
-* ``dns.resourceRecordSets.update``
+In most cases, configuring credentials for Certbot will require `creating a service account
+<https://cloud.google.com/iam/docs/service-accounts-create>`_, and then either `granting permissions
+with predefined roles`_ or `granting permissions with custom roles`_ using IAM.
 
-(The closest role is `dns.admin <https://cloud.google.com/dns/docs/
-access-control#dns.admin>`_).
 
-If the above permissions are assigned at the `resource level <https://cloud
-.google.com/dns/docs/zones/iam-per-resource-zones>`_, the same user must
-have, at the PROJECT level, the following permissions:
+Providing Credentials
+^^^^^^^^^^^^^^^^^^^^^
 
-* ``dns.managedZones.get``
-* ``dns.managedZones.list``
+The preferred method of providing credentials is to `set up Application Default Credentials
+<https://cloud.google.com/docs/authentication/provide-credentials-adc>`_ (ADC) in the environment
+that Certbot is running in.
 
-(The closest role is `dns.reader <https://cloud.google.com/dns/docs/
-access-control#dns.reader>`_).
+If you are running Certbot on Google Cloud then a service account can be assigned directly to most
+types of workload, including `Compute Engine VMs <https://cloud.google.com/compute/docs/access/
+create-enable-service-accounts-for-instances>`_, `Kubernetes Engine Pods <https://cloud.google.com/
+kubernetes-engine/docs/how-to/workload-identity>`_, `Cloud Run jobs <https://cloud.google.com/run
+/docs/securing/service-identity>`_, `Cloud Functions <https://cloud.google.com/functions/docs/
+securing/function-identity>`_, and `Cloud Builds <https://cloud.google.com/build/docs/securing-
+builds/configure-user-specified-service-accounts>`_.
 
-Google provides instructions for `creating a service account <https://developers
-.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount>`_ and
-`information about the required permissions <https://cloud.google.com/dns/access
--control#permissions_and_roles>`_. If you're running on Google Compute Engine,
-you can `assign the service account to the instance <https://cloud.google.com/
-compute/docs/access/create-enable-service-accounts-for-instances>`_ which
-is running certbot. A credentials file is not required in this case, as they
-are automatically obtained by certbot through the `metadata service
-<https://cloud.google.com/compute/docs/storing-retrieving-metadata>`_ .
+If you are not running Certbot on Google Cloud then a credentials file should be provided using the
+``--dns-google-credentials`` command-line argument. Google provides documentation for `creating
+service account keys <https://cloud.google.com/iam/docs/keys-create-delete#creating>`_, which is the
+most common method of using a service account outside of Google Cloud.
 
 .. code-block:: json
-   :name: credentials.json
-   :caption: Example credentials file:
+   :name: credentials-sa.json
+   :caption: Example service account key file:
 
    {
       "type": "service_account",
@@ -78,12 +77,8 @@
       "client_x509_cert_url": "..."
    }
 
-The path to this file can be provided interactively or using the
-``--dns-google-credentials`` command-line argument. Certbot records the path
-to this file for use during renewal, but does not store the file's contents.
-
 .. caution::
-   You should protect these API credentials as you would a password. Users who
+   You should protect these credentials as you would a password. Users who
    can read this file can use these credentials to issue some types of API calls
    on your behalf, limited by the permissions assigned to the account. Users who
    can cause Certbot to run using these credentials can complete a ``dns-01``
@@ -97,35 +92,132 @@
 including for renewal, and cannot be silenced except by addressing the issue
 (e.g., by using a command like ``chmod 600`` to restrict access to the file).
 
+If you are running Certbot within another cloud platform, a CI platform, or any other platform that
+supports issuing OpenID Connect Tokens, then you may also have the option of securely authenticating
+with `workload identity federation <https://cloud.google.com/iam/docs/workload-identity-
+federation>`_. Instructions are generally available for most platforms, including `AWS or Azure
+<https://cloud.google.com/iam/docs/workload-identity-federation-with-other-clouds>`_, `GitHub
+Actions <https://cloud.google.com/blog/products/identity-security/enabling-keyless-authentication
+-from-github-actions>`_, and `GitLab CI <https://docs.gitlab.com/ee/ci/cloud_services/
+google_cloud/>`_.
+
+
+Granting Permissions with Predefined Roles
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The simplest method of granting the required permissions to the user or service account that Certbot
+is authenticating with is to use either of these predefined role strategies:
+
+* `dns.admin <https://cloud.google.com/dns/docs/access-control#dns.admin>`_ against the *DNS
+  zone(s)* that Certbot will be issuing certificates for.
+* `dns.reader <https://cloud.google.com/dns/docs/access-control#dns.reader>`_ against the *project*
+  containing the relevant DNS zones.
+
+*or*
+
+* `dns.admin <https://cloud.google.com/dns/docs/access-control#dns.admin>`_ against the *project*
+  containing the relevant DNS zones
+
+For instructions on how to grant roles, please read the Google provided documentation for `granting
+access roles against a project <https://cloud.google.com/iam/docs/granting-changing-revoking-access
+#single-role>`_ and `granting access roles against zones <https://cloud.google.com/dns/docs/zones/
+iam-per-resource-zones#set_access_control_policy_for_a_specific_resource>`_.
+
+.. caution::
+   Granting the ``dns.admin`` role at the project level can present a significant security risk. It
+   provides full administrative access to all DNS zones within the project, granting the ability to
+   perform any action up to and including deleting all zones within a project.
+
+
+Granting Permissions with Custom Roles
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Custom roles are an alternative to predefined roles that provide the ability to define fine grained
+permission sets for specific use cases. They should generally be used when it is desirable to adhere
+to the principle of least privilege, such as within production or other security sensitive
+workloads.
+
+The following is an example strategy for granting granular permissions to Certbot using custom
+roles. If you are not already familiar with how to do so, Google provides documentation for
+`creating a custom IAM role <https://cloud.google.com/iam/docs/creating-custom-roles#creating>`_.
+
+Firstly, create a custom role containing the permissions required to make DNS record updates. We
+suggest naming the custom role ``Certbot - Zone Editor`` with the ID ``certbot.zoneEditor``. The
+following permissions are required:
+
+* ``dns.changes.create``
+* ``dns.changes.get``
+* ``dns.changes.list``
+* ``dns.resourceRecordSets.create``
+* ``dns.resourceRecordSets.delete``
+* ``dns.resourceRecordSets.list``
+* ``dns.resourceRecordSets.update``
+
+Next, create a custom role granting Certbot the ability to discover DNS zones. We suggest naming the
+custom role ``Certbot - Zone Lister`` with the ID ``certbot.zoneLister``. The following permissions
+are required:
+
+* ``dns.managedZones.get``
+* ``dns.managedZones.list``
+
+Finally, grant the custom roles to the user or service account that Certbot is authenticating with:
+
+* Grant your custom ``Certbot - Zone Editor`` role against the *DNS zone(s)* that Certbot will be
+  issuing certificates for.
+* Grant your custom ``Certbot - Zone Lister`` role against the *project* containing the relevant DNS
+  zones.
+
+For instructions on how to grant roles, please read the Google provided documentation for `granting
+access roles against a project <https://cloud.google.com/iam/docs/granting-changing-revoking-access
+#single-role>`_ and `granting access roles against zones <https://cloud.google.com/dns/docs/zones/
+iam-per-resource-zones#set_access_control_policy_for_a_specific_resource>`_.
+
 
 Examples
 --------
 
 .. code-block:: bash
-   :caption: To acquire a certificate for ``example.com``
+   :caption: To acquire a certificate for ``example.com``, providing a credentials file
 
    certbot certonly \\
      --dns-google \\
      --dns-google-credentials ~/.secrets/certbot/google.json \\
      -d example.com
 
+.. code-block:: bash
+   :caption: To acquire a certificate for ``example.com``, where ADC is available and
+             a credentials file is not required
+
+   certbot certonly \\
+     --dns-google \\
+     -d example.com
+
 .. code-block:: bash
    :caption: To acquire a single certificate for both ``example.com`` and
              ``www.example.com``
 
    certbot certonly \\
      --dns-google \\
-     --dns-google-credentials ~/.secrets/certbot/google.json \\
      -d example.com \\
      -d www.example.com
 
+.. code-block:: bash
+   :caption: To acquire a certificate for ``example.com``, where the managed DNS
+             zone resides in another Google Cloud project.
+
+   certbot certonly \\
+     --dns-google \\
+     --dns-google-credentials ~/.secrets/certbot/google-project-test-foo.json \\
+     --dns-google-project test-bar \\
+     -d example.com
+
+
 .. code-block:: bash
    :caption: To acquire a certificate for ``example.com``, waiting 120 seconds
              for DNS propagation
 
    certbot certonly \\
      --dns-google \\
-     --dns-google-credentials ~/.secrets/certbot/google.json \\
      --dns-google-propagation-seconds 120 \\
      -d example.com
 
diff --git a/certbot-dns-google/certbot_dns_google/_internal/dns_google.py b/certbot-dns-google/certbot_dns_google/_internal/dns_google.py
--- a/certbot-dns-google/certbot_dns_google/_internal/dns_google.py
+++ b/certbot-dns-google/certbot_dns_google/_internal/dns_google.py
@@ -1,21 +1,22 @@
 """DNS Authenticator for Google Cloud DNS."""
-import json
 import logging
 from typing import Any
 from typing import Callable
 from typing import Dict
 from typing import Optional
 
+import google.auth
+
+from google.auth import exceptions as googleauth_exceptions
 from googleapiclient import discovery
 from googleapiclient import errors as googleapiclient_errors
-import httplib2
-from oauth2client.service_account import ServiceAccountCredentials
 
 from certbot import errors
 from certbot.plugins import dns_common
 
 logger = logging.getLogger(__name__)
 
+ADC_URL = 'https://cloud.google.com/docs/authentication/application-default-credentials'
 ACCT_URL = 'https://developers.google.com/identity/protocols/OAuth2ServiceAccount#creatinganaccount'
 PERMISSIONS_URL = 'https://cloud.google.com/dns/access-control#permissions_and_roles'
 METADATA_URL = 'http://metadata.google.internal/computeMetadata/v1/'
@@ -31,15 +32,23 @@ class Authenticator(dns_common.DNSAuthenticator):
     description = ('Obtain certificates using a DNS TXT record (if you are using Google Cloud DNS '
                    'for DNS).')
     ttl = 60
+    google_client = None
 
     @classmethod
     def add_parser_arguments(cls, add: Callable[..., None],
                              default_propagation_seconds: int = 60) -> None:
         super().add_parser_arguments(add, default_propagation_seconds=60)
         add('credentials',
-            help=('Path to Google Cloud DNS service account JSON file. (See {0} for' +
-                  'information about creating a service account and {1} for information about the' +
-                  'required permissions.)').format(ACCT_URL, PERMISSIONS_URL),
+            help=('Path to Google Cloud DNS service account JSON file to use instead of relying on'
+                  ' Application Default Credentials (ADC). (See {0} for information about ADC, {1}'
+                  ' for information about creating a service account, and {2} for information about'
+                  ' the permissions required to modify Cloud DNS records.)')
+                  .format(ADC_URL, ACCT_URL, PERMISSIONS_URL),
+            default=None)
+
+        add('project',
+            help=('The ID of the Google Cloud project that the Google Cloud DNS managed zone(s)' +
+                  ' reside in. This will be determined automatically if not specified.'),
             default=None)
 
     def more_info(self) -> str:
@@ -47,22 +56,19 @@ def more_info(self) -> str:
                'the Google Cloud DNS API.'
 
     def _setup_credentials(self) -> None:
-        if self.conf('credentials') is None:
-            try:
-                # use project_id query to check for availability of google metadata server
-                # we won't use the result but know we're not on GCP when an exception is thrown
-                _GoogleClient.get_project_id()
-            except (ValueError, httplib2.ServerNotFoundError):
-                raise errors.PluginError('Unable to get Google Cloud Metadata and no credentials'
-                                         ' specified. Automatic credential lookup is only '
-                                         'available on Google Cloud Platform. Please configure'
-                                         ' credentials using --dns-google-credentials <file>')
-        else:
+        if self.conf('credentials') is not None:
             self._configure_file('credentials',
                                  'path to Google Cloud DNS service account JSON file')
 
             dns_common.validate_file_permissions(self.conf('credentials'))
 
+        try:
+            self._get_google_client()
+        except googleauth_exceptions.DefaultCredentialsError as e:
+            raise errors.PluginError('Authentication using Google Application Default Credentials '
+                                     'failed ({}). Please configure credentials using'
+                                     ' --dns-google-credentials <file>'.format(e))
+
     def _perform(self, domain: str, validation_name: str, validation: str) -> None:
         self._get_google_client().add_txt_record(domain, validation_name, validation, self.ttl)
 
@@ -70,7 +76,10 @@ def _cleanup(self, domain: str, validation_name: str, validation: str) -> None:
         self._get_google_client().del_txt_record(domain, validation_name, validation, self.ttl)
 
     def _get_google_client(self) -> '_GoogleClient':
-        return _GoogleClient(self.conf('credentials'))
+        if self.google_client is None:
+            self.google_client = _GoogleClient(self.conf('credentials'), self.conf('project'))
+        return self.google_client
+
 
 
 class _GoogleClient:
@@ -79,20 +88,31 @@ class _GoogleClient:
     """
 
     def __init__(self, account_json: Optional[str] = None,
+                 dns_project_id: Optional[str] = None,
                  dns_api: Optional[discovery.Resource] = None) -> None:
 
         scopes = ['https://www.googleapis.com/auth/ndev.clouddns.readwrite']
+        credentials = None
+        project_id = None
+
         if account_json is not None:
             try:
-                credentials = ServiceAccountCredentials.from_json_keyfile_name(account_json, scopes)
-                with open(account_json) as account:
-                    self.project_id = json.load(account)['project_id']
-            except Exception as e:
+                credentials, project_id = google.auth.load_credentials_from_file(
+                    account_json, scopes=scopes)
+            except googleauth_exceptions.GoogleAuthError as e:
                 raise errors.PluginError(
-                    "Error parsing credentials file '{}': {}".format(account_json, e))
+                    "Error loading credentials file '{}': {}".format(account_json, e))
         else:
-            credentials = None
-            self.project_id = self.get_project_id()
+            credentials, project_id = google.auth.default(scopes=scopes)
+
+        if dns_project_id is not None:
+            project_id = dns_project_id
+
+        if not project_id:
+            raise errors.PluginError('The Google Cloud project could not be automatically '
+                                     'determined. Please configure it using --dns-google-project'
+                                     ' <project>.')
+        self.project_id = project_id
 
         if not dns_api:
             self.dns = discovery.build('dns', 'v1',
@@ -292,26 +312,3 @@ def _find_managed_zone_id(self, domain: str) -> str:
 
         raise errors.PluginError('Unable to determine managed zone for {0} using zone names: {1}.'
                                  .format(domain, zone_dns_name_guesses))
-
-    @staticmethod
-    def get_project_id() -> str:
-        """
-        Query the google metadata service for the current project ID
-
-        This only works on Google Cloud Platform
-
-        :raises ServerNotFoundError: Not running on Google Compute or DNS not available
-        :raises ValueError: Server is found, but response code is not 200
-        :returns: project id
-        """
-        url = '{0}project/project-id'.format(METADATA_URL)
-
-        # Request an access token from the metadata server.
-        http = httplib2.Http()
-        r, content = http.request(url, headers=METADATA_HEADERS)
-        if r.status != 200:
-            raise ValueError("Invalid status code: {0}".format(r))
-
-        if isinstance(content, bytes):
-            return content.decode()
-        return content
diff --git a/certbot-dns-google/setup.py b/certbot-dns-google/setup.py
--- a/certbot-dns-google/setup.py
+++ b/certbot-dns-google/setup.py
@@ -7,11 +7,9 @@
 version = '2.6.0.dev0'
 
 install_requires = [
-    'google-api-python-client>=1.5.5',
-    'oauth2client>=4.0',
+    'google-api-python-client>=1.6.5',
+    'google-auth>=2.16.0',
     'setuptools>=41.6.0',
-    # already a dependency of google-api-python-client, but added for consistency
-    'httplib2'
 ]
 
 if not os.environ.get('SNAP_BUILD'):


Test patch:
diff --git a/certbot-dns-google/certbot_dns_google/_internal/tests/dns_google_test.py b/certbot-dns-google/certbot_dns_google/_internal/tests/dns_google_test.py
--- a/certbot-dns-google/certbot_dns_google/_internal/tests/dns_google_test.py
+++ b/certbot-dns-google/certbot_dns_google/_internal/tests/dns_google_test.py
@@ -4,10 +4,10 @@
 import unittest
 from unittest import mock
 
+from google.auth import exceptions as googleauth_exceptions
 from googleapiclient import discovery
 from googleapiclient.errors import Error
 from googleapiclient.http import HttpMock
-from httplib2 import ServerNotFoundError
 import pytest
 
 from certbot import errors
@@ -20,7 +20,7 @@
 ACCOUNT_JSON_PATH = '/not/a/real/path.json'
 API_ERROR = Error()
 PROJECT_ID = "test-test-1"
-
+SCOPES = ['https://www.googleapis.com/auth/ndev.clouddns.readwrite']
 
 class AuthenticatorTest(test_util.TempDirTestCase, dns_test_common.BaseAuthenticatorTest):
 
@@ -34,22 +34,25 @@ def setUp(self):
 
         super().setUp()
         self.config = mock.MagicMock(google_credentials=path,
+                                     google_project=PROJECT_ID,
                                      google_propagation_seconds=0)  # don't wait during tests
 
         self.auth = Authenticator(self.config, "google")
 
         self.mock_client = mock.MagicMock()
-        # _get_google_client | pylint: disable=protected-access
-        self.auth._get_google_client = mock.MagicMock(return_value=self.mock_client)
 
     @test_util.patch_display_util()
     def test_perform(self, unused_mock_get_utility):
+        # _get_google_client | pylint: disable=protected-access
+        self.auth._get_google_client = mock.MagicMock(return_value=self.mock_client)
         self.auth.perform([self.achall])
 
         expected = [mock.call.add_txt_record(DOMAIN, '_acme-challenge.'+DOMAIN, mock.ANY, mock.ANY)]
         assert expected == self.mock_client.mock_calls
 
     def test_cleanup(self):
+        # _get_google_client | pylint: disable=protected-access
+        self.auth._get_google_client = mock.MagicMock(return_value=self.mock_client)
         # _attempt_cleanup | pylint: disable=protected-access
         self.auth._attempt_cleanup = True
         self.auth.cleanup([self.achall])
@@ -57,13 +60,27 @@ def test_cleanup(self):
         expected = [mock.call.del_txt_record(DOMAIN, '_acme-challenge.'+DOMAIN, mock.ANY, mock.ANY)]
         assert expected == self.mock_client.mock_calls
 
-    @mock.patch('httplib2.Http.request', side_effect=ServerNotFoundError)
     @test_util.patch_display_util()
-    def test_without_auth(self, unused_mock_get_utility, unused_mock):
+    def test_without_auth(self, unused_mock_get_utility):
+        self.auth._get_google_client = mock.MagicMock(side_effect=googleauth_exceptions.DefaultCredentialsError)
         self.config.google_credentials = None
         with pytest.raises(PluginError):
             self.auth.perform([self.achall])
 
+    @mock.patch('certbot_dns_google._internal.dns_google._GoogleClient')
+    def test_get_google_client(self, client_mock):
+        test_client = mock.MagicMock()
+        client_mock.return_value = test_client
+
+        self.auth._get_google_client()
+        assert client_mock.called
+        assert self.auth.google_client is test_client
+
+    def test_get_google_client_cached(self):
+        test_client = mock.MagicMock()
+        self.auth.google_client = test_client
+        assert self.auth._get_google_client() is test_client
+
 
 class GoogleClientTest(unittest.TestCase):
     record_name = "foo"
@@ -81,7 +98,7 @@ def _setUp_client_with_mock(self, zone_request_side_effect, rrs_list_side_effect
         http_mock = HttpMock(discovery_file, {'status': '200'})
         dns_api = discovery.build('dns', 'v1', http=http_mock)
 
-        client = _GoogleClient(ACCOUNT_JSON_PATH, dns_api)
+        client = _GoogleClient(ACCOUNT_JSON_PATH, None, dns_api)
 
         # Setup
         mock_mz = mock.MagicMock()
@@ -107,32 +124,67 @@ def rrs_list(project=None, managedZone=None, name=None, type=None):
         return client, mock_changes
 
     @mock.patch('googleapiclient.discovery.build')
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
-    @mock.patch('certbot_dns_google._internal.dns_google._GoogleClient.get_project_id')
-    def test_client_without_credentials(self, get_project_id_mock, credential_mock,
-                                        unused_discovery_mock):
+    @mock.patch('google.auth.default')
+    def test_client_with_default_credentials(self, credential_mock, discovery_mock):
+        test_credentials = mock.MagicMock()
+        credential_mock.return_value = (test_credentials, PROJECT_ID)
         from certbot_dns_google._internal.dns_google import _GoogleClient
-        _GoogleClient(None)
-        assert not credential_mock.called
-        assert get_project_id_mock.called
+        client = _GoogleClient(None)
+        credential_mock.assert_called_once_with(scopes=SCOPES)
+        assert client.project_id == PROJECT_ID
+        discovery_mock.assert_called_once_with('dns', 'v1',
+                                credentials=test_credentials,
+                                cache_discovery=False)
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('googleapiclient.discovery.build')
+    @mock.patch('google.auth.load_credentials_from_file')
+    def test_client_with_json_credentials(self, credential_mock, discovery_mock):
+        test_credentials = mock.MagicMock()
+        credential_mock.return_value = (test_credentials, PROJECT_ID)
+        from certbot_dns_google._internal.dns_google import _GoogleClient
+        client = _GoogleClient(ACCOUNT_JSON_PATH)
+        credential_mock.assert_called_once_with(ACCOUNT_JSON_PATH, scopes=SCOPES)
+        assert credential_mock.called
+        assert client.project_id == PROJECT_ID
+        discovery_mock.assert_called_once_with('dns', 'v1',
+                                       credentials=test_credentials,
+                                       cache_discovery=False)
+
+    @mock.patch('google.auth.load_credentials_from_file')
     def test_client_bad_credentials_file(self, credential_mock):
-        credential_mock.side_effect = ValueError('Some exception buried in oauth2client')
+        credential_mock.side_effect = googleauth_exceptions.DefaultCredentialsError('Some exception buried in google.auth')
         with pytest.raises(errors.PluginError) as exc_info:
             self._setUp_client_with_mock([])
         assert str(exc_info.value) == \
-            "Error parsing credentials file '/not/a/real/path.json': " \
-            "Some exception buried in oauth2client"
+            "Error loading credentials file '/not/a/real/path.json': " \
+            "Some exception buried in google.auth"
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
+    def test_client_missing_project_id(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), "")
+        with pytest.raises(errors.PluginError) as exc_info:
+            self._setUp_client_with_mock([])
+        assert str(exc_info.value) == \
+            "The Google Cloud project could not be automatically determined. " \
+            "Please configure it using --dns-google-project <project>."
+
+    @mock.patch('googleapiclient.discovery.build')
+    @mock.patch('google.auth.default')
+    def test_client_with_project_id(self, credential_mock, unused_discovery_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+        from certbot_dns_google._internal.dns_google import _GoogleClient
+        client = _GoogleClient(None, "test-project-2")
+        assert credential_mock.called
+        assert client.project_id == "test-project-2"
+
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    @mock.patch('certbot_dns_google._internal.dns_google._GoogleClient.get_project_id')
-    def test_add_txt_record(self, get_project_id_mock, credential_mock):
+    def test_add_txt_record(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}])
-        credential_mock.assert_called_once_with('/not/a/real/path.json', mock.ANY)
-        assert not get_project_id_mock.called
+        credential_mock.assert_called_once_with('/not/a/real/path.json', scopes=SCOPES)
 
         client.add_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
@@ -153,10 +205,12 @@ def test_add_txt_record(self, get_project_id_mock, credential_mock):
                                                managedZone=self.zone,
                                                project=PROJECT_ID)
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_add_txt_record_and_poll(self, unused_credential_mock):
+    def test_add_txt_record_and_poll(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}])
         changes.create.return_value.execute.return_value = {'status': 'pending', 'id': self.change}
         changes.get.return_value.execute.return_value = {'status': 'done'}
@@ -171,10 +225,12 @@ def test_add_txt_record_and_poll(self, unused_credential_mock):
                                             managedZone=self.zone,
                                             project=PROJECT_ID)
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_add_txt_record_delete_old(self, unused_credential_mock):
+    def test_add_txt_record_delete_old(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock(
             [{'managedZones': [{'id': self.zone}]}])
         # pylint: disable=line-too-long
@@ -187,10 +243,12 @@ def test_add_txt_record_delete_old(self, unused_credential_mock):
             assert "sample-txt-contents" in deletions["rrdatas"]
             assert self.record_ttl == deletions["ttl"]
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_add_txt_record_delete_old_ttl_case(self, unused_credential_mock):
+    def test_add_txt_record_delete_old_ttl_case(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock(
             [{'managedZones': [{'id': self.zone}]}])
         # pylint: disable=line-too-long
@@ -204,49 +262,59 @@ def test_add_txt_record_delete_old_ttl_case(self, unused_credential_mock):
             assert "sample-txt-contents" in deletions["rrdatas"]
             assert custom_ttl == deletions["ttl"] #otherwise HTTP 412
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_add_txt_record_noop(self, unused_credential_mock):
+    def test_add_txt_record_noop(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock(
             [{'managedZones': [{'id': self.zone}]}])
         client.add_txt_record(DOMAIN, "_acme-challenge.example.org",
                               "example-txt-contents", self.record_ttl)
         assert changes.create.called is False
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_add_txt_record_error_during_zone_lookup(self, unused_credential_mock):
+    def test_add_txt_record_error_during_zone_lookup(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, unused_changes = self._setUp_client_with_mock(API_ERROR)
 
         with pytest.raises(errors.PluginError):
             client.add_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_add_txt_record_zone_not_found(self, unused_credential_mock):
+    def test_add_txt_record_zone_not_found(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, unused_changes = self._setUp_client_with_mock([{'managedZones': []},
                                                                {'managedZones': []}])
 
         with pytest.raises(errors.PluginError):
             client.add_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_add_txt_record_error_during_add(self, unused_credential_mock):
+    def test_add_txt_record_error_during_add(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}])
         changes.create.side_effect = API_ERROR
 
         with pytest.raises(errors.PluginError):
             client.add_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_del_txt_record_multi_rrdatas(self, unused_credential_mock):
+    def test_del_txt_record_multi_rrdatas(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}])
         # pylint: disable=line-too-long
         mock_get_rrs = "certbot_dns_google._internal.dns_google._GoogleClient.get_existing_txt_rrset"
@@ -282,10 +350,12 @@ def test_del_txt_record_multi_rrdatas(self, unused_credential_mock):
                                                managedZone=self.zone,
                                                project=PROJECT_ID)
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_del_txt_record_single_rrdatas(self, unused_credential_mock):
+    def test_del_txt_record_single_rrdatas(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}])
         # pylint: disable=line-too-long
         mock_get_rrs = "certbot_dns_google._internal.dns_google._GoogleClient.get_existing_txt_rrset"
@@ -311,36 +381,44 @@ def test_del_txt_record_single_rrdatas(self, unused_credential_mock):
                                                managedZone=self.zone,
                                                project=PROJECT_ID)
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_del_txt_record_error_during_zone_lookup(self, unused_credential_mock):
+    def test_del_txt_record_error_during_zone_lookup(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock(API_ERROR)
         client.del_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
         changes.create.assert_not_called()
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_del_txt_record_zone_not_found(self, unused_credential_mock):
+    def test_del_txt_record_zone_not_found(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock([{'managedZones': []},
                                                                {'managedZones': []}])
         client.del_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
         changes.create.assert_not_called()
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_del_txt_record_error_during_delete(self, unused_credential_mock):
+    def test_del_txt_record_error_during_delete(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, changes = self._setUp_client_with_mock([{'managedZones': [{'id': self.zone}]}])
         changes.create.side_effect = API_ERROR
 
         client.del_txt_record(DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_get_existing_found(self, unused_credential_mock):
+    def test_get_existing_found(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, unused_changes = self._setUp_client_with_mock(
             [{'managedZones': [{'id': self.zone}]}])
         # Record name mocked in setUp
@@ -348,69 +426,39 @@ def test_get_existing_found(self, unused_credential_mock):
         assert found["rrdatas"] == ["\"example-txt-contents\""]
         assert found["ttl"] == 60
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_get_existing_not_found(self, unused_credential_mock):
+    def test_get_existing_not_found(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, unused_changes = self._setUp_client_with_mock(
             [{'managedZones': [{'id': self.zone}]}])
         not_found = client.get_existing_txt_rrset(self.zone, "nonexistent.tld")
         assert not_found is None
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_get_existing_with_error(self, unused_credential_mock):
+    def test_get_existing_with_error(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, unused_changes = self._setUp_client_with_mock(
             [{'managedZones': [{'id': self.zone}]}], API_ERROR)
         # Record name mocked in setUp
         found = client.get_existing_txt_rrset(self.zone, "_acme-challenge.example.org")
         assert found is None
 
-    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('google.auth.load_credentials_from_file')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_get_existing_fallback(self, unused_credential_mock):
+    def test_get_existing_fallback(self, credential_mock):
+        credential_mock.return_value = (mock.MagicMock(), PROJECT_ID)
+
         client, unused_changes = self._setUp_client_with_mock(
             [{'managedZones': [{'id': self.zone}]}], API_ERROR)
         rrset = client.get_existing_txt_rrset(self.zone, "_acme-challenge.example.org")
         assert not rrset
 
-    def test_get_project_id(self):
-        from certbot_dns_google._internal.dns_google import _GoogleClient
-
-        response = DummyResponse()
-        response.status = 200
-
-        with mock.patch('httplib2.Http.request', return_value=(response, 'test-test-1')):
-            project_id = _GoogleClient.get_project_id()
-            assert project_id == 'test-test-1'
-
-        with mock.patch('httplib2.Http.request', return_value=(response, b'test-test-1')):
-            project_id = _GoogleClient.get_project_id()
-            assert project_id == 'test-test-1'
-
-        failed_response = DummyResponse()
-        failed_response.status = 404
-
-        with mock.patch('httplib2.Http.request',
-                        return_value=(failed_response, "some detailed http error response")):
-            with pytest.raises(ValueError):
-                _GoogleClient.get_project_id()
-
-        with mock.patch('httplib2.Http.request', side_effect=ServerNotFoundError):
-            with pytest.raises(ServerNotFoundError):
-                _GoogleClient.get_project_id()
-
-
-class DummyResponse:
-    """
-    Dummy object to create a fake HTTPResponse (the actual one requires a socket and we only
-     need the status attribute)
-    """
-    def __init__(self):
-        self.status = 200
-
-
 if __name__ == "__main__":
     sys.exit(pytest.main(sys.argv[1:] + [__file__]))  # pragma: no cover