Repository: sunpy/sunpy

Issue:
Automate updating vendored code in `sunpy/extern`
### Describe the feature

I'm currently manually checking for updates of vendored code in `/sunpy/extern`. It should be pretty easy to automate this in a simple script in the `/tools` folder that downloads the latest version of the files, and prints which files have been updated (and possibly the version number bumps?)

### Proposed solution

_No response_


Discussion:
Hi @dstansby, I would like to work on this.
I don't understand which part of `/sunpy/extern` must be automated, can you guide me through this?
All that I understood is, that we are going to automate the process of updating versions, with the help of a script that will be in `/tools`. 
I have a question, when will this script in `/tools` will be executed?
> Hi @dstansby, I would like to work on this. I don't understand which part of `/sunpy/extern` must be automated, can you guide me through this? All that I understood is, that we are going to automate the process of updating versions, with the help of a script that will be in `/tools`.

Yes that is correct, what would you like to know in addition to this? 

> I have a question, when will this script in `/tools` will be executed?

It will be run by the person doing a sunpy release. 

which files in `/extern` should be automated?
> which files in `/extern` should be automated?

Every python file expect `modest_image.py`, I don't think there is a proper source to update it. The licences should be fine to leave alone. There is a case we should keep them in sync if they have changed but we can leave that out for the first version of the script.
Automate these changes?
https://github.com/sunpy/sunpy/commit/5ba9c9141e3364529138f8451225a97bb195851d
https://github.com/sunpy/sunpy/commit/9f5427cc4939db023a36092b6bb43cd3505249d8
https://github.com/sunpy/sunpy/commit/9ab27122dd96b69f88831c54106ed251f6bdee09

These commits also have changes that are different from changing versions (I mean, some changes change function), how are we going to do that?

What should I use to download the latest files and check the versions?
> Automate these changes? [5ba9c91](https://github.com/sunpy/sunpy/commit/5ba9c9141e3364529138f8451225a97bb195851d) [9f5427c](https://github.com/sunpy/sunpy/commit/9f5427cc4939db023a36092b6bb43cd3505249d8) [9ab2712](https://github.com/sunpy/sunpy/commit/9ab27122dd96b69f88831c54106ed251f6bdee09)
> 

Yes to create a script that will for the packages we have in the `sunpy/extern` folder, update them from their canonical source.

> These commits also have changes that are different from changing versions (I mean, some changes change function), how are we going to do that?

I don't follow. 

> What should I use to download the latest files and check the versions?

Whatever works. You probably want to check with PyPi, or you could check the latest tags on GitHub. 

> Yes to create a script that will for the packages we have in the sunpy/extern folder, update them from their canonical source.

Any pointers on how to start?
> > Yes to create a script that will for the packages we have in the sunpy/extern folder, update them from their canonical source.
> 
> Any pointers on how to start?

List out what packages need updating, work out the best way to get the Python code (probably PyPi would be easier), download the source code, extract it and copy it to the `sunpy/extern` folder.   

Does that help? Or did you want something different? 
> List out what packages need updating

For this we can use `pip list --outdated`

which gives a list of packages that can be updated.
```py
Package    Version Latest Type
---------- ------- ------ -----
matplotlib 3.5.1   3.5.2  wheel
setuptools 61.3.1  62.1.0 wheel
```

> download the source code

Do you mean, downloading the `source distribution` of the package?

For example in matplotlib
![Screenshot from 2022-05-04 13-59-40](https://user-images.githubusercontent.com/53405133/166647279-8ff20ac6-27f8-42a6-91da-36ffcc30b4f6.png)

In the above example, Do we need to download `matplotlib-3.5.2.tar.gz`?
> > List out what packages need updating
> 
> For this we can use `pip list --outdated`
> 
> which gives a list of packages that can be updated.
> 
> ```python
> Package    Version Latest Type
> ---------- ------- ------ -----
> matplotlib 3.5.1   3.5.2  wheel
> setuptools 61.3.1  62.1.0 wheel
> ```

We only want the packages that exist inside the folder: `sunpy/extern` (https://github.com/sunpy/sunpy/tree/main/sunpy/extern)

> > download the source code
> 
> Do you mean, downloading the `source distribution` of the package?
> In the above example, Do we need to download `matplotlib-3.5.2.tar.gz`?

You would have two choices, either to work out the latest version of the package on pypi, then download the source distribution, extract it, copy out the file we need.

Or you use the GitHub repository, get the latest tag and download the file directly.

I am unsure what would be easier to do. 
> packages that exist inside the folder: `sunpy/extern`

Do you mean, all the packages that were imported by the files in `sunpy/extern` (except `modest_image.py`)?
- https://github.com/sunpy/sunpy/blob/main/sunpy/extern/distro.py#L31
- https://github.com/sunpy/sunpy/blob/main/sunpy/extern/inflect.py#L52
- https://github.com/sunpy/sunpy/blob/main/sunpy/extern/parse.py#L475


Using github API would be a good solution. we would have to make GET request at `https://api.github.com/repos/{org_name}/{repo_name}/releases/latest`
For example, lets get latest version of matplotlib:
```sh
 ~ curl "https://api.github.com/repos/matplotlib/matplotlib/releases/latest" 
{
  "url": "https://api.github.com/repos/matplotlib/matplotlib/releases/65879732",
  "assets_url": "https://api.github.com/repos/matplotlib/matplotlib/releases/65879732/assets",
  "upload_url": "https://uploads.github.com/repos/matplotlib/matplotlib/releases/65879732/assets{?name,label}",
  "html_url": "https://github.com/matplotlib/matplotlib/releases/tag/v3.5.2",
  "id": 65879732,
  "author": {
    "login": "QuLogic",
    "id": 302469,
    "node_id": "MDQ6VXNlcjMwMjQ2OQ==",
    "avatar_url": "https://avatars.githubusercontent.com/u/302469?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/QuLogic",
    "html_url": "https://github.com/QuLogic",
    "followers_url": "https://api.github.com/users/QuLogic/followers",
    "following_url": "https://api.github.com/users/QuLogic/following{/other_user}",
    "gists_url": "https://api.github.com/users/QuLogic/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/QuLogic/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/QuLogic/subscriptions",
    "organizations_url": "https://api.github.com/users/QuLogic/orgs",
    "repos_url": "https://api.github.com/users/QuLogic/repos",
    "events_url": "https://api.github.com/users/QuLogic/events{/privacy}",
    "received_events_url": "https://api.github.com/users/QuLogic/received_events",
    "type": "User",
    "site_admin": false
  },
  "node_id": "RE_kwDOABUios4D7T60",
  "tag_name": "v3.5.2",
  "target_commitish": "main",
  "name": "REL: v3.5.2",
  "draft": false,
  "prerelease": false,
  "created_at": "2022-05-03T03:50:24Z",
  "published_at": "2022-05-03T03:58:04Z",
  "assets": [

  ],
  "tarball_url": "https://api.github.com/repos/matplotlib/matplotlib/tarball/v3.5.2",
  "zipball_url": "https://api.github.com/repos/matplotlib/matplotlib/zipball/v3.5.2",
  "body": "This is the second bugfix release of the 3.5.x series.\r\n\r\nThis release contains several bug-fixes and adjustments:\r\n\r\n* Add support for Windows on ARM (source-only; no wheels provided yet)\r\n* Add year to concise date formatter when displaying less than 12 months\r\n* Disable `QuadMesh` mouse cursor to avoid severe performance regression in `pcolormesh`\r\n* Delay backend selection to allow choosing one in more cases\r\n* Fix automatic layout bugs in EPS output\r\n* Fix autoscaling of `scatter` plots\r\n* Fix clearing of subfigures\r\n* Fix colorbar exponents, inversion of extensions, and use on inset axes\r\n* Fix compatibility with various NumPy-like classes (e.g., Pandas, xarray, etc.)\r\n* Fix constrained layout bugs with mixed subgrids\r\n* Fix `errorbar` with dashes\r\n* Fix errors in conversion to GTK4 and Qt6\r\n* Fix figure options accidentally re-ordering data\r\n* Fix keyboard focus of TkAgg backend\r\n* Fix manual selection of contour labels\r\n* Fix path effects on text with whitespace\r\n* Fix `quiver` in subfigures\r\n* Fix `RangeSlider.set_val` displaying incorrectly\r\n* Fix regressions in collection data limits\r\n* Fix `stairs` with no edgecolor\r\n* Fix some leaks in Tk backends\r\n* Fix tight layout DPI confusion\r\n* Fix tool button customizability and some tool manager bugs\r\n* Only set Tk HiDPI scaling-on-map for Windows systems\r\n* Partially allow TTC font collection files by selecting the first font",
  "reactions": {
    "url": "https://api.github.com/repos/matplotlib/matplotlib/releases/65879732/reactions",
    "total_count": 9,
    "+1": 2,
    "-1": 0,
    "laugh": 0,
    "hooray": 3,
    "confused": 0,
    "heart": 0,
    "rocket": 4,
    "eyes": 0
  }
}
```
And we need value of this key in the json file:
`"tag_name": "v3.5.2"`

I will soon submit a PR, thanks for the help. 
> > packages that exist inside the folder: `sunpy/extern`
> 
> Do you mean, all the packages that were imported by the files in `sunpy/extern` (except `modest_image.py`)?
> 
> * [https://github.com/sunpy/sunpy/blob/main/sunpy/extern/distro.py#L31](https://github.com/sunpy/sunpy/blob/main/sunpy/extern/distro.py?rgh-link-date=2022-05-04T16%3A16%3A23Z#L31)
> * [https://github.com/sunpy/sunpy/blob/main/sunpy/extern/inflect.py#L52](https://github.com/sunpy/sunpy/blob/main/sunpy/extern/inflect.py?rgh-link-date=2022-05-04T16%3A16%3A23Z#L52)
> * [https://github.com/sunpy/sunpy/blob/main/sunpy/extern/parse.py#L475](https://github.com/sunpy/sunpy/blob/main/sunpy/extern/parse.py?rgh-link-date=2022-05-04T16%3A16%3A23Z#L475)

Yes.

> Using github API would be a good solution. we would have to make GET request at `https://api.github.com/repos/{org_name}/{repo_name}/releases/latest` For example, lets get latest version of matplotlib:
> 
> ```shell
>  ~ curl "https://api.github.com/repos/matplotlib/matplotlib/releases/latest" 
> {
>   "url": "https://api.github.com/repos/matplotlib/matplotlib/releases/65879732",
>   "assets_url": "https://api.github.com/repos/matplotlib/matplotlib/releases/65879732/assets",
>   "upload_url": "https://uploads.github.com/repos/matplotlib/matplotlib/releases/65879732/assets{?name,label}",
>   "html_url": "https://github.com/matplotlib/matplotlib/releases/tag/v3.5.2",
>   "id": 65879732,
>   "author": {
>     "login": "QuLogic",
>     "id": 302469,
>     "node_id": "MDQ6VXNlcjMwMjQ2OQ==",
>     "avatar_url": "https://avatars.githubusercontent.com/u/302469?v=4",
>     "gravatar_id": "",
>     "url": "https://api.github.com/users/QuLogic",
>     "html_url": "https://github.com/QuLogic",
>     "followers_url": "https://api.github.com/users/QuLogic/followers",
>     "following_url": "https://api.github.com/users/QuLogic/following{/other_user}",
>     "gists_url": "https://api.github.com/users/QuLogic/gists{/gist_id}",
>     "starred_url": "https://api.github.com/users/QuLogic/starred{/owner}{/repo}",
>     "subscriptions_url": "https://api.github.com/users/QuLogic/subscriptions",
>     "organizations_url": "https://api.github.com/users/QuLogic/orgs",
>     "repos_url": "https://api.github.com/users/QuLogic/repos",
>     "events_url": "https://api.github.com/users/QuLogic/events{/privacy}",
>     "received_events_url": "https://api.github.com/users/QuLogic/received_events",
>     "type": "User",
>     "site_admin": false
>   },
>   "node_id": "RE_kwDOABUios4D7T60",
>   "tag_name": "v3.5.2",
>   "target_commitish": "main",
>   "name": "REL: v3.5.2",
>   "draft": false,
>   "prerelease": false,
>   "created_at": "2022-05-03T03:50:24Z",
>   "published_at": "2022-05-03T03:58:04Z",
>   "assets": [
> 
>   ],
>   "tarball_url": "https://api.github.com/repos/matplotlib/matplotlib/tarball/v3.5.2",
>   "zipball_url": "https://api.github.com/repos/matplotlib/matplotlib/zipball/v3.5.2",
>   "body": "This is the second bugfix release of the 3.5.x series.\r\n\r\nThis release contains several bug-fixes and adjustments:\r\n\r\n* Add support for Windows on ARM (source-only; no wheels provided yet)\r\n* Add year to concise date formatter when displaying less than 12 months\r\n* Disable `QuadMesh` mouse cursor to avoid severe performance regression in `pcolormesh`\r\n* Delay backend selection to allow choosing one in more cases\r\n* Fix automatic layout bugs in EPS output\r\n* Fix autoscaling of `scatter` plots\r\n* Fix clearing of subfigures\r\n* Fix colorbar exponents, inversion of extensions, and use on inset axes\r\n* Fix compatibility with various NumPy-like classes (e.g., Pandas, xarray, etc.)\r\n* Fix constrained layout bugs with mixed subgrids\r\n* Fix `errorbar` with dashes\r\n* Fix errors in conversion to GTK4 and Qt6\r\n* Fix figure options accidentally re-ordering data\r\n* Fix keyboard focus of TkAgg backend\r\n* Fix manual selection of contour labels\r\n* Fix path effects on text with whitespace\r\n* Fix `quiver` in subfigures\r\n* Fix `RangeSlider.set_val` displaying incorrectly\r\n* Fix regressions in collection data limits\r\n* Fix `stairs` with no edgecolor\r\n* Fix some leaks in Tk backends\r\n* Fix tight layout DPI confusion\r\n* Fix tool button customizability and some tool manager bugs\r\n* Only set Tk HiDPI scaling-on-map for Windows systems\r\n* Partially allow TTC font collection files by selecting the first font",
>   "reactions": {
>     "url": "https://api.github.com/repos/matplotlib/matplotlib/releases/65879732/reactions",
>     "total_count": 9,
>     "+1": 2,
>     "-1": 0,
>     "laugh": 0,
>     "hooray": 3,
>     "confused": 0,
>     "heart": 0,
>     "rocket": 4,
>     "eyes": 0
>   }
> }
> ```
> 
> And we need value of this key in the json file: `"tag_name": "v3.5.2"`

The only issue here is what if they don't do a formal GitHub release, what happens then?

Resolving patch:
diff --git a/tools/update_extern.py b/tools/update_extern.py
new file mode 100644
--- /dev/null
+++ b/tools/update_extern.py
@@ -0,0 +1,53 @@
+"""
+Updates all the libraries in ``sunpy/extern``
+"""
+
+import json
+import urllib.request
+from pathlib import Path
+
+SUNPY_EXTERN_DIR = Path(__file__).parent.parent / "sunpy" / "extern"
+
+# "filename": ["user", "repository", "path_to_file"]
+PACKAGES = {
+    "appdirs.py": ["ActiveState", "appdirs", "appdirs.py"],
+    "appdirs_license.txt": ["ActiveState", "appdirs", "LICENSE.txt"],
+    "distro.py": ["python-distro", "distro", "src/distro/distro.py"],
+    "distro_license.rst": ["python-distro", "distro", "LICENSE"],
+    "inflect.py": ["jaraco", "inflect", "inflect/__init__.py"],
+    "inflect_license.txt": ["jaraco", "inflect", "LICENSE"],
+    "parse.py": ["r1chardj0n3s", "parse", "parse.py"],
+    "parse_license.txt": ["r1chardj0n3s", "parse", "LICENSE"],
+}
+
+
+def download_github_file(user: str, repo: str, src: Path, dest: Path):
+    """
+    Download a file from Github.
+    """
+    try:
+        response = urllib.request.urlopen(f"https://api.github.com/repos/{user}/{repo}")
+    except urllib.error.HTTPError as e:
+        print(f"Error: {e}")
+        return
+
+    url = f"https://api.github.com/repos/{user}/{repo}/tags"
+    response = urllib.request.urlopen(url)
+    if response.status != 200:
+        raise ValueError(f"tags for {user}/{repo} does not exist.")
+    response = json.load(response)
+    version = response[0]["name"]
+
+    url = f"https://raw.githubusercontent.com/{user}/{repo}/{version}/{src}"
+    response = urllib.request.urlopen(url)
+    if response.status != 200:
+        raise ValueError(f"{url} does not exist.")
+
+    with open(dest, "wb") as f:
+        print(f"Updating {dest} to {user}/{repo}:refs/tags/{version}")
+        f.write(response.read())
+
+
+if __name__ == "__main__":
+    for package, (user, repo, src) in PACKAGES.items():
+        download_github_file(user, repo, src, SUNPY_EXTERN_DIR / package)


Test patch: