Repository: biolab/orange3

Issue:
File can not read Google Sheets
<!-- 
Thanks for taking the time to report a bug!
If you're raising an issue about an add-on (i.e., installed via Options > Add-ons), raise an issue in the relevant add-on's issue tracker instead. See: https://github.com/biolab?q=orange3
To fix the bug, we need to be able to reproduce it. Please answer the following questions to the best of your ability.
-->


**What's wrong?**
<!-- Be specific, clear, and concise. Include screenshots if relevant. -->
<!-- If you're getting an error message, copy it, and enclose it with three backticks (```). -->
Google Drive apparently changed the API to retrieve Google Sheets. Now, one cannot just enter sharable link to load the data (Read error).

We should check the API and see whether we can fix this so that one can easily load the data from the link.


**How can we reproduce the problem?** 
<!-- Upload a zip with the .ows file and data. -->
<!-- Describe the steps (open this widget, click there, then add this...) -->
Try to load any data from Google Sheets via link.




**What's your environment?**
<!-- To find your Orange version, see "Help → About → Version" or `Orange.version.full_version` in code -->
- Operating system: OSX
- Orange version: 3.30.2dev
- How you installed Orange: pip



Discussion:
The issue here is that since https://github.com/biolab/orange3/commit/62a7605ba4c08db89703838ec9e767cc400a7a1a we encode URLs, and google docs services don't like this.

Plain URL works fine: https://docs.google.com/spreadsheets/d/1PJhmH0X_N7hZn03le5J70E4IKL4LiqAZenlhg2kWWUU/edit#gid=0

Encoded URL cannot be found:
https://docs.google.com/spreadsheets/d/1PJhmH0X_N7hZn03le5J70E4IKL4LiqAZenlhg2kWWUU/edit%23gid%3D0

This seems to be specific to google services because Dropbox links and links to biolab servers work fine:
http://file.biolab.si/text-semantics/data/elektrotehniski-vestnik-clanki/detektiranje-utrdb-v-%C5%A1ahu-.txt

One solution is to do an additional check for 'docs.google.com' URLs and skip encoding entirely. Any other ideas?







Hi, Lukas from AlgOps.com here.

This issue is probably not based on API because in older versions of Orange (3.27 respectively) it worked even though it didn't work in newer versions. A

@JakaKokosar also your plain URL doesn't work fine - I tried this several minutes ago and it returns the same error even though I am able to look at the Sheet in the browser.

Please can you speed up on this issue since a big portion of our customers can't use both AlgOps and Orange due to this?

Thanks!
The bug is there from merging #5412. So this bug may stem from:
- `quote` being overzealous, at least in that URL part - I do not know the exact URL specification, but quote is written as a general function which does not distinguish between parts of the URL
- dropbox and google docs have a special functions `_trim_*` functions that in docs' case adds a sheet (maybe something else) - maybe quoting prevents it

@lukasholovsky, if the functionality is important for you, consider making a PR.

Resolving patch:
diff --git a/Orange/data/io.py b/Orange/data/io.py
--- a/Orange/data/io.py
+++ b/Orange/data/io.py
@@ -406,7 +406,13 @@ def __init__(self, filename):
         filename = filename.strip()
         if not urlparse(filename).scheme:
             filename = 'http://' + filename
-        filename = quote(filename, safe="/:")
+
+        # Fully support URL with query or fragment like http://filename.txt?a=1&b=2#c=3
+        def quote_byte(b):
+            return chr(b) if b < 0x80 else '%{:02X}'.format(b)
+
+        filename = ''.join(map(quote_byte, filename.encode("utf-8")))
+
         super().__init__(filename)
 
     @staticmethod


Test patch:
diff --git a/Orange/tests/test_url_reader.py b/Orange/tests/test_url_reader.py
--- a/Orange/tests/test_url_reader.py
+++ b/Orange/tests/test_url_reader.py
@@ -25,6 +25,19 @@ def test_special_characters(self):
                "vestnik-clanki/detektiranje-utrdb-v-šahu-.txt"
         self.assertRaises(OSError, UrlReader(path).read)
 
+    def test_base_url_with_query(self):
+        data = UrlReader("https://datasets.biolab.si/core/grades.xlsx?a=1&b=2").read()
+        self.assertEqual(16, len(data))
+
+    def test_url_with_fragment(self):
+        data = UrlReader("https://datasets.biolab.si/core/grades.xlsx#tab=1").read()
+        self.assertEqual(16, len(data))
+
+    def test_special_characters_with_query_and_fragment(self):
+        path = "http://file.biolab.si/text-semantics/data/elektrotehniski-" \
+               "vestnik-clanki/detektiranje-utrdb-v-šahu-.txt?a=1&b=2#c=3"
+        self.assertRaises(OSError, UrlReader(path).read)
+
 
 if __name__ == "__main__":
     unittest.main()