Repository: kedro-org/kedro

Issue:
`APIDataSet` ease of use
## Description

Howdy team!

I was working with the `APIDataSet` recently and had two issues out of the box.

#### 1. Specifying the `auth` keyword argument in yaml

The `requests` library expects the `auth` parameter of a request to be either a `HTTPBasicAuth` or a `tuple` (lists are not allowed, see [here](https://github.com/ianwhale/kedro-kaggle-starter/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/%7B%7B%20cookiecutter.python_package%20%7D%7D/api.py) in requests). At the moment, neither are possible to specify in my `catalog.yml`. 

From what I hear, you're already working on this (#1011). So maybe this point is moot.

#### 2. The `auth` keyword argument and `credentials.yml`

I would like to specify my `(username, password)` tuple inside `credentials.yml`. However, the `APIDataSet`'s `auth` keyword wouldn't get filled in by the config loader. 

To get this working, you'd have to extend `APIDataSet` to have a `credentials` keyword that is filled in for `auth` in an upcall.

It would be great to either have this by default, or even have the loader fill `auth` keywords in addition to `credentials`. Although that might have unintended consequences. 

## Context

Hopefully this would unify the experience a bit. Right now, the `credentials` keyword in a dataset and `credentials.yml` are the main points of access to secrets. Which is probably good.

## Possible Implementation

I whipped up [my own `APIDataSet`](https://github.com/ianwhale/kedro-kaggle-starter/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/%7B%7B%20cookiecutter.python_package%20%7D%7D/api.py) to solve both the problems above.

## Possible Alternatives

To get this working with no changes to `APIDataSet`, we'd have to implement the changes in #1011 so we can specify tuples in `credentials.yml` and have the config loader fill in `auth` as well.


Discussion:
Hi @ianwhale, thank you for your very thoughtful write up of this. The points you raise are very fair. I'd suggest just implementing the changes you propose to `APIDataSet` right away as it will give an instant improvement, seems pretty uncontroversial and I like the code exactly how you wrote it 👍 

For the possible alternative: #1011 is still very much up for discussion and would be a larger change. And `auth` is only relevant to the `APIDataSet` so I'd be inclined to leave the config loader alone.

Resolving patch:
diff --git a/kedro/extras/datasets/api/api_dataset.py b/kedro/extras/datasets/api/api_dataset.py
--- a/kedro/extras/datasets/api/api_dataset.py
+++ b/kedro/extras/datasets/api/api_dataset.py
@@ -1,7 +1,7 @@
 """``APIDataSet`` loads the data from HTTP(S) APIs.
 It uses the python requests library: https://requests.readthedocs.io/en/master/
 """
-from typing import Any, Dict, List, Tuple, Union
+from typing import Any, Dict, Iterable, List, Union
 
 import requests
 from requests.auth import AuthBase
@@ -41,9 +41,10 @@ def __init__(
         data: Any = None,
         params: Dict[str, Any] = None,
         headers: Dict[str, Any] = None,
-        auth: Union[Tuple[str], AuthBase] = None,
+        auth: Union[Iterable[str], AuthBase] = None,
         json: Union[List, Dict[str, Any]] = None,
         timeout: int = 60,
+        credentials: Union[Iterable[str], AuthBase] = None,
     ) -> None:
         """Creates a new instance of ``APIDataSet`` to fetch data from an API endpoint.
 
@@ -57,15 +58,29 @@ def __init__(
             headers: The HTTP headers.
                 https://requests.readthedocs.io/en/master/user/quickstart/#custom-headers
             auth: Anything ``requests`` accepts. Normally it's either ``('login', 'password')``,
-                or ``AuthBase``, ``HTTPBasicAuth`` instance for more complex cases.
+                or ``AuthBase``, ``HTTPBasicAuth`` instance for more complex cases. Any
+                iterable will be cast to a tuple.
             json: The request payload, used for POST, PUT, etc requests, passed in
                 to the json kwarg in the requests object.
                 https://requests.readthedocs.io/en/master/user/quickstart/#more-complicated-post-requests
             timeout: The wait time in seconds for a response, defaults to 1 minute.
                 https://requests.readthedocs.io/en/master/user/quickstart/#timeouts
+            credentials: same as ``auth``. Allows specifying ``auth`` secrets in
+                credentials.yml.
 
+        Raises:
+            ValueError: if both ``credentials`` and ``auth`` are specified.
         """
         super().__init__()
+
+        if credentials is not None and auth is not None:
+            raise ValueError("Cannot specify both auth and credentials.")
+
+        auth = credentials or auth
+
+        if isinstance(auth, Iterable):
+            auth = tuple(auth)
+
         self._request_args: Dict[str, Any] = {
             "url": url,
             "method": method,


Test patch:
diff --git a/tests/extras/datasets/api/test_api_dataset.py b/tests/extras/datasets/api/test_api_dataset.py
--- a/tests/extras/datasets/api/test_api_dataset.py
+++ b/tests/extras/datasets/api/test_api_dataset.py
@@ -127,3 +127,44 @@ def test_exists_ok(self, requests_mocker, method):
         )
 
         assert api_data_set.exists()
+
+    def test_credentials_auth_error(self, method):
+        """
+        If ``auth`` and ``credentials`` are both provided,
+        the constructor should raise a ValueError.
+        """
+        with pytest.raises(ValueError, match="both auth and credentials"):
+            APIDataSet(url=TEST_URL, method=method, auth=[], credentials=[])
+
+    @pytest.mark.parametrize("auth_kwarg", ["auth", "credentials"])
+    @pytest.mark.parametrize(
+        "auth_seq",
+        [
+            ("username", "password"),
+            ["username", "password"],
+            (e for e in ["username", "password"]),  # Generator.
+        ],
+    )
+    def test_auth_sequence(self, requests_mocker, method, auth_seq, auth_kwarg):
+        """
+        ``auth`` and ``credentials`` should be able to be any Iterable.
+        """
+        kwargs = {
+            "url": TEST_URL,
+            "method": method,
+            "params": TEST_PARAMS,
+            "headers": TEST_HEADERS,
+            auth_kwarg: auth_seq,
+        }
+
+        api_data_set = APIDataSet(**kwargs)
+        requests_mocker.register_uri(
+            method,
+            TEST_URL_WITH_PARAMS,
+            headers=TEST_HEADERS,
+            text=TEST_TEXT_RESPONSE_DATA,
+        )
+
+        response = api_data_set.load()
+        assert isinstance(response, requests.Response)
+        assert response.text == TEST_TEXT_RESPONSE_DATA