Repository: iterative/dvc

Issue:
dvc pull -r http_remote: fails with 401 - unauthorized for directory cache files, when basic auth is set properly
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug. 

Example: `repro: doesn't detect input changes`
-->

## Description
When running dvc pull for an HTTP cache protected with basic auth, directory files read cause:
```
ERROR: unexpected error - 401 Client Error: Unauthorized for url: http://<remote_url>/c2/f5094a84cd17aeeca0524bd422b5ab.dir
```

<!--
A clear and concise description of what the bug is.
-->

### Reproduce
* Git clone repository with a DVC HTTP remote protected with basic auth
* Repository should have a directory tracked by DVC.
* Configure basic auth
   ```
    dvc remote add <remote_name> http://<remote_url>
    dvc remote modify origin --local auth basic
    dvc remote modify origin --local user <user>
    dvc remote modify origin --local password <password>
    ```
* Run DVC pull -r <remote_name>
<!--
Step list of how to reproduce the bug
-->

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

### Expected
Pull all the files from the cache, including the directory and its content.
<!--
A clear and concise description of what you expect to happen.
-->

### Environment information
* MacOS 11.4
* DVC 2.6.4 / 2.6.0
* HTTP remote - Private repository on DAGsHub
<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
DVC version: 2.6.4
---------------------------------
Platform: Python 3.7.7 on Darwin-20.5.0-x86_64-i386-64bit
Supports:
	gdrive (pydrive2 = 1.7.0),
	hdfs (pyarrow = 2.0.0),
	webhdfs (hdfs = 2.5.8),
	http (requests = 2.24.0),
	https (requests = 2.24.0)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: http, http
Workspace directory: apfs on /dev/disk1s2s1
Repo: dvc, git
```

**Additional Information (if any):**
The origin of the issue seems to be in file `dvc/objects/tree.py` in the function `load` in line 106.
This line ultimately leads to an HTTP request to be done without any `auth` specified in the `open` method of the class `BaseFileSystem`, when it calls `open_url`

<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->



Discussion:
Possibly related?
#5677 

Resolving patch:
diff --git a/dvc/fs/base.py b/dvc/fs/base.py
--- a/dvc/fs/base.py
+++ b/dvc/fs/base.py
@@ -9,7 +9,6 @@
 from dvc.progress import Tqdm
 from dvc.utils import tmp_fname
 from dvc.utils.fs import makedirs, move
-from dvc.utils.http import open_url
 
 logger = logging.getLogger(__name__)
 
@@ -126,12 +125,6 @@ def _check_requires(self, **kwargs):
         )
 
     def open(self, path_info, mode: str = "r", encoding: str = None, **kwargs):
-        if hasattr(self, "_generate_download_url"):
-            # pylint:disable=no-member
-            func = self._generate_download_url  # type: ignore[attr-defined]
-            get_url = partial(func, path_info)
-            return open_url(get_url, mode=mode, encoding=encoding, **kwargs)
-
         raise RemoteActionNotImplemented("open", self.scheme)
 
     def exists(self, path_info) -> bool:
diff --git a/dvc/fs/http.py b/dvc/fs/http.py
--- a/dvc/fs/http.py
+++ b/dvc/fs/http.py
@@ -65,9 +65,6 @@ def _auth_method(self, url):
                 self.headers.update({self.custom_auth_header: self.password})
         return None
 
-    def _generate_download_url(self, path_info):
-        return path_info.url
-
     @wrap_prop(threading.Lock())
     @cached_property
     def _session(self):
@@ -190,6 +187,17 @@ def _upload(
                 desc=name or to_info.url,
             )
 
+    def open(self, path_info, mode: str = "r", encoding: str = None, **kwargs):
+        from dvc.utils.http import open_url
+
+        return open_url(
+            path_info.url,
+            mode=mode,
+            encoding=encoding,
+            auth=self._auth_method(path_info),
+            **kwargs,
+        )
+
     @staticmethod
     def _content_length(response) -> Optional[int]:
         res = response.headers.get("Content-Length")
diff --git a/dvc/utils/http.py b/dvc/utils/http.py
--- a/dvc/utils/http.py
+++ b/dvc/utils/http.py
@@ -24,13 +24,15 @@ def open_url(url, mode="r", encoding=None, **iter_opts):
 
 
 @contextmanager
-def iter_url(url, chunk_size=io.DEFAULT_BUFFER_SIZE):
+def iter_url(url, chunk_size=io.DEFAULT_BUFFER_SIZE, auth=None):
     """Iterate over chunks requested from url."""
     import requests
 
     def request(headers=None):
         the_url = url() if callable(url) else url
-        response = requests.get(the_url, stream=True, headers=headers)
+        response = requests.get(
+            the_url, stream=True, headers=headers, auth=auth
+        )
         if response.status_code == 404:
             raise FileNotFoundError(f"Can't open {the_url}")
         response.raise_for_status()


Test patch: