Repository: kivy/python-for-android

Issue:
Error (Downloading matplotlib from https://jqueryui.com/resources/download/jquery-ui-1.12.1.zip)
<!--
The issue tracker is a tool to address bugs NOT a support platform.
Please use the Discord community or Stack Overflow for support questions,
more information at https://github.com/kivy/python-for-android#support
-->

### Checklist

- [x] the issue is indeed a bug and not a support request
- [x] issue doesn't already exist: https://github.com/kivy/python-for-android/issues
- [x] I have a short, runnable example that reproduces the issue
- [x] I reproduced the problem with the latest development version (`p4a.branch = develop`)
- [x] I used the grave accent (aka backticks) to format code or logs when appropriated

### Versions

- Python:3.8.5
- OS:Ubuntu 20.04
- Kivy:2.0.0
- Cython:0.29.21
- OpenJDK:8u282-b08-0ubuntu1~20.04

### Description

// REPLACE ME: What are you trying to get done, what has happened, what went wrong, and what did you expect?

Error occurred while downloading https://jqueryui.com/resources/download/jquery-ui-1.12.1.zip (for matplotlib)

### buildozer.spec


Command:
``` buildozer android debug deploy run

// REPLACE ME: buildozer command ran? e.g. buildozer android debug
// Keep the triple grave accent (aka backquote/backtick) to have the code formatted
```

Spec file:
```
....
requirements = kivy==2.0.0, android, plyer, python3, matplotlib
....
```

### Logs

```
..........
ESC[1m[INFO]ESC[0m:    Downloading matplotlib from https://jqueryui.com/resources/download/jquery-ui-1.12.1.zip
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/pano/kivy_buildozer/apps/kivyLauncher/github20210212/kivy-launcher/python-for-android/pythonforandroid/toolchain.py", line 1266, in <mod
ule>
    main()
  File "/home/pano/kivy_buildozer/apps/kivyLauncher/github20210212/kivy-launcher/python-for-android/pythonforandroid/entrypoints.py", line 18, in main
    ToolchainCL()
  File "/home/pano/kivy_buildozer/apps/kivyLauncher/github20210212/kivy-launcher/python-for-android/pythonforandroid/toolchain.py", line 715, in __ini
t__

    getattr(self, command)(args)
  File "/home/pano/kivy_buildozer/apps/kivyLauncher/github20210212/kivy-launcher/python-for-android/pythonforandroid/toolchain.py", line 153, in wrapp
er_func
    build_dist_from_args(ctx, dist, args)
  File "/home/pano/kivy_buildozer/apps/kivyLauncher/github20210212/kivy-launcher/python-for-android/pythonforandroid/toolchain.py", line 212, in build
_dist_from_args
    build_recipes(build_order, python_modules, ctx,
  File "/home/pano/kivy_buildozer/apps/kivyLauncher/github20210212/kivy-launcher/python-for-android/pythonforandroid/build.py", line 571, in build_rec
ipes
    recipe.prebuild_arch(arch)
  File "/home/pano/kivy_buildozer/apps/kivyLauncher/github20210212/kivy-launcher/python-for-android/pythonforandroid/recipes/matplotlib/__init__.py", 
line 104, in prebuild_arch
    self.download_web_backend_dependencies(arch)
  File "/home/pano/kivy_buildozer/apps/kivyLauncher/github20210212/kivy-launcher/python-for-android/pythonforandroid/recipes/matplotlib/__init__.py", 
line 93, in download_web_backend_dependencies
    self.download_file(url, target_file)
  File "/home/pano/kivy_buildozer/apps/kivyLauncher/github20210212/kivy-launcher/python-for-android/pythonforandroid/recipe.py", line 209, in download
_file
    urlretrieve(url, target, report_hook)
  File "/usr/lib/python3.8/urllib/request.py", line 247, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
..........
```

I was able to resolve the error with the following changes.
****patch
```
diff -uprN python-for-android.org/pythonforandroid/recipe.py python-for-android/pythonforandroid/recipe.py
--- python-for-android.org/pythonforandroid/recipe.py	2021-02-12 21:32:25.927590713 +0900
+++ python-for-android/pythonforandroid/recipe.py	2021-02-12 21:33:15.696658685 +0900
@@ -10,6 +10,7 @@ import sh
 import shutil
 import fnmatch
 from urllib.request import urlretrieve
+import urllib
 from os import listdir, unlink, environ, mkdir, curdir, walk
 from sys import stdout
 import time
@@ -206,6 +207,9 @@ class Recipe(with_metaclass(RecipeMeta))
             seconds = 1
             while True:
                 try:
+                    opener = urllib.request.build_opener()
+                    opener.addheaders = [('User-agent', 'Mozilla/5.0')]
+                    urllib.request.install_opener(opener)
                     urlretrieve(url, target, report_hook)
                 except OSError as e:
                     attempts += 1
```
****patch


Discussion:
Huh. It looks like `jqueryui.com` is specifically blocking any request with a user-agent that includes the string "Python-urllib".

I suggest you open a pull request to get that patch merged :)

Resolving patch:
diff --git a/pythonforandroid/recipe.py b/pythonforandroid/recipe.py
--- a/pythonforandroid/recipe.py
+++ b/pythonforandroid/recipe.py
@@ -9,6 +9,7 @@
 import sh
 import shutil
 import fnmatch
+import urllib.request
 from urllib.request import urlretrieve
 from os import listdir, unlink, environ, mkdir, curdir, walk
 from sys import stdout
@@ -23,6 +24,11 @@
 from pythonforandroid.util import load_source as import_recipe
 
 
+url_opener = urllib.request.build_opener()
+url_orig_headers = url_opener.addheaders
+urllib.request.install_opener(url_opener)
+
+
 class RecipeMeta(type):
     def __new__(cls, name, bases, dct):
         if name != 'Recipe':
@@ -206,6 +212,8 @@ def report_hook(index, blksize, size):
             seconds = 1
             while True:
                 try:
+                    # jqueryui.com returns a 403 w/ the default user agent
+                    url_opener.addheaders = [('User-agent', 'Mozilla/5.0')]
                     urlretrieve(url, target, report_hook)
                 except OSError as e:
                     attempts += 1
@@ -215,6 +223,8 @@ def report_hook(index, blksize, size):
                     time.sleep(seconds)
                     seconds *= 2
                     continue
+                finally:
+                    url_opener.addheaders = url_orig_headers
                 break
             return target
         elif parsed_url.scheme in ('git', 'git+file', 'git+ssh', 'git+http', 'git+https'):


Test patch: