Repository: spotify/luigi

Issue:
using default_scheduler_url as a mounting point (not root `http://address/mount`)  behind proxy not working
<!---
We use GitHub issues mainly for tracking bugs and feature requests.
Questions for how to use luigi can be sent to the mailing list.

Currently, there are no strict procedures or guidelines for submitting issues.
In short, please just use common sense.

Common sense includes this at bare-minimum:

 * search for similar issues posted before creating a new issue.
 * Use markdown to format all code/logs. Issues which are hard to read
   when rendered on GitHub might be closed with a friendly reminder of this.
 * If applicable, reading relevant parts of the documentation.

Also, add steps to reproduce the bug, if applicable. Sample code would be nice too :)

For more information on how to submit valuable contributions,
see https://opensource.guide/how-to-contribute/#how-to-submit-a-contribution
-->
I need to mount `luigi` behind a nginx, for example luigi at `http://address/mount`.
For that I configure:
```ini
[core]
default_scheduler_url=http://address/mount
....
```

GUI is ok and works but, CLI not due to url resolution. it happens there https://github.com/spotify/luigi/blob/c13566418c92de3e4d8d33ead4e7c936511afae1/luigi/rpc.py#L54

To understand what happened:
```python
parsed=urlparse('http://address/mount')
url='/api/add_task'
urljoin(parsed.geturl(), url)
# ==> give 'http://address/api/add_task'
# expected http://address/mount/api/add_task
```
What I must do for working - slash at the end of mount point, no slash for url -:
```python
parsed=urlparse('http://address/mount/')
url='api/add_task'
urljoin(parsed.geturl(), url)
# ==> http://address/mount/api/add_task
```


Discussion:

Resolving patch:
diff --git a/luigi/rpc.py b/luigi/rpc.py
--- a/luigi/rpc.py
+++ b/luigi/rpc.py
@@ -55,11 +55,12 @@ def _urljoin(base, url):
     """
     Join relative URLs to base URLs like urllib.parse.urljoin but support
     arbitrary URIs (esp. 'http+unix://').
+    base part is fixed or mounted point, every url contains full base part.
     """
     parsed = urlparse(base)
     scheme = parsed.scheme
     return urlparse(
-        urljoin(parsed._replace(scheme='http').geturl(), url)
+        urljoin(parsed._replace(scheme='http').geturl(), parsed.path + (url if url[0] == '/' else '/' + url))
     )._replace(scheme=scheme).geturl()
 
 


Test patch:
diff --git a/test/rpc_test.py b/test/rpc_test.py
--- a/test/rpc_test.py
+++ b/test/rpc_test.py
@@ -38,6 +38,14 @@ def testUrlArgumentVariations(self):
                     s._fetch(suffix, '{}')
                     fetcher.fetch.assert_called_once_with('http://zorg.com/api/123', '{}', 42)
 
+    def testUrlArgumentVariationsNotRoot(self):
+        for url in ['http://zorg.com/subpath', 'http://zorg.com/subpath/']:
+            for suffix in ['api/123', '/api/123']:
+                s = luigi.rpc.RemoteScheduler(url, 42)
+                with mock.patch.object(s, '_fetcher') as fetcher:
+                    s._fetch(suffix, '{}')
+                    fetcher.fetch.assert_called_once_with('http://zorg.com/subpath/api/123', '{}', 42)
+
     def get_work(self, fetcher_side_effect):
         scheduler = luigi.rpc.RemoteScheduler('http://zorg.com', 42)
         scheduler._rpc_retry_wait = 1  # shorten wait time to speed up tests