Repository: Cadair/parfive

Issue:
Propagate the URL which is associated with each file path through to the Results object
It would be useful if the `Results` object had a property which mapped input URL to output filename, as the filename is normally provided by the `Content-Disposition` headers on the download request, so is not known to the user at the point they call download.

Discussion:
samaloney: The doc build is real seems to need at least `sphinx>=5` but have it pinned to `sphinx<5`. I've removed this and fixed the resulting error in this PR too but can pull it out to a separate one?
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/Cadair/parfive/pull/146?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Stuart+Mumford) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 90.24%. Comparing base [(`3b049c5`)](https://app.codecov.io/gh/Cadair/parfive/commit/3b049c55f5cd35a607aafdfecfbccabf7bc71e29?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Stuart+Mumford) to head [(`97ee1ea`)](https://app.codecov.io/gh/Cadair/parfive/pull/146?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Stuart+Mumford).
> Report is 2 commits behind head on main.


<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #146      +/-   ##
==========================================
+ Coverage   90.23%   90.24%   +0.01%     
==========================================
  Files           5        5              
  Lines         635      646      +11     
==========================================
+ Hits          573      583      +10     
- Misses         62       63       +1     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/Cadair/parfive/pull/146?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Stuart+Mumford).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Stuart+Mumford).

samaloney: I'm not sure about the test fail it passes with py37 on my Mac which is running Sonoma
Cadair: @samaloney can you rebase? I pulled your documentation fixes into #149 
Cadair: oh no more conflicts, sorry!

Patch:
diff --git a/parfive/downloader.py b/parfive/downloader.py
index 540d2b6..4b3a6fe 100644
--- a/parfive/downloader.py
+++ b/parfive/downloader.py
@@ -319,7 +319,8 @@ class Downloader:
             elif isinstance(res, Exception):
                 raise res
             else:
-                results.append(res)
+                requested_url, filepath = res
+                results.append(path=filepath, url=requested_url)
 
         return results
 
@@ -551,7 +552,8 @@ class Downloader:
                             "File %s already exists and overwrite is False; skipping download.",
                             filepath,
                         )
-                        return str(filepath)
+                        return url, str(filepath)
+
                     if callable(file_pb):
                         file_pb = file_pb(
                             position=token.n,
@@ -618,9 +620,11 @@ class Downloader:
             await asyncio.gather(*tasks)
             # join() waits till all the items in the queue have been processed
             await downloaded_chunk_queue.join()
+
             for callback in self.config.done_callbacks:
                 callback(filepath, url, None)
-            return str(filepath)
+
+            return url, str(filepath)
 
         except (Exception, asyncio.CancelledError) as e:
             for task in tasks:
@@ -810,7 +814,7 @@ class Downloader:
                         "File %s already exists and overwrite is False; skipping download.",
                         filepath,
                     )
-                    return str(filepath)
+                    return url, str(filepath)
 
                 if callable(file_pb):
                     total_size = await get_ftp_size(client, parse.path)
@@ -845,7 +849,7 @@ class Downloader:
                     for callback in self.config.done_callbacks:
                         callback(filepath, url, None)
 
-                    return str(filepath)
+                    return url, str(filepath)
 
         except (Exception, asyncio.CancelledError) as e:
             if writer is not None:
diff --git a/parfive/results.py b/parfive/results.py
index 390622a..e4c718c 100644
--- a/parfive/results.py
+++ b/parfive/results.py
@@ -22,14 +22,16 @@ class Results(UserList):
     """
     The results of a download from `parfive.Downloader.download`.
 
-    This object contains the filenames of successful downloads as well
-    as a list of any errors encountered in the `~parfive.Results.errors`
+    This object contains the filenames of successful downloads as well,
+    a list of all urls requested in the `~parfive.Results.urls` property
+    and a list of any errors encountered in the `~parfive.Results.errors`
     property.
     """
 
-    def __init__(self, *args, errors=None):
+    def __init__(self, *args, errors=None, urls=None):
         super().__init__(*args)
         self._errors = errors or list()
+        self._urls = urls or list()
 
     def _get_nice_resp_repr(self, response):
         # This is a modified version of aiohttp.ClientResponse.__repr__
@@ -63,6 +65,10 @@ class Results(UserList):
         out += str(self)
         return out
 
+    def append(self, *, path, url):
+        super().append(path)
+        self._urls.append(url)
+
     def add_error(self, filename, url, exception):
         """
         Add an error to the results.
@@ -82,3 +88,11 @@ class Results(UserList):
         ``exception`` is the error raised during download.
         """
         return self._errors
+
+    @property
+    def urls(self):
+        """
+        A list of requested urls.
+
+        """
+        return self._urls


Test patch:
diff --git a/parfive/tests/test_downloader.py b/parfive/tests/test_downloader.py
index 3cfd255..99c6f4f 100644
--- a/parfive/tests/test_downloader.py
+++ b/parfive/tests/test_downloader.py
@@ -47,6 +47,7 @@ def test_download(httpserver, tmpdir):
     assert dl.queued_downloads == 1
 
     f = dl.download()
+    f.urls == [httpserver.url]
     validate_test_file(f)
 
 
@@ -302,7 +303,10 @@ def test_failed_download():
 def test_results():
     res = Results()
 
-    res.append("hello")
+    res.append(path="hello", url="aurl")
+
+    assert res[0] == "hello"
+    assert res.urls[0] == "aurl"
 
     res.add_error("wibble", "notaurl", "out of cheese")