Repository: apache/airflow

Issue:
Make Docs builds fallback in case external docs sources are missing
Every now and then our docs builds start to fail because of external dependency (latest example here #14985). And while we are doing caching now of that information, it does not help when the initial retrieval fails. This information does not change often but with the number of dependencies we have it will continue to fail regularly simply because many of those depenencies are not very reliable - they are just a web page hosted somewhere. They are nowhere near the stabilty of even PyPI or Apt sources and we have no mirroring in case of problem.

Maybe we could 

a) see if we can use some kind of mirroring scheme (do those sites have mirrrors ? )
b) if not, simply write a simple script that will dump the cached content for those to S3, refresh it in the CI scheduled (nightly) master builds ad have a fallback mechanism to download that from there in case of any problems in CI?

 


Discussion:
Looks like our sphinx can handle it, but mirroring seems to be a  feature of intersphinx mapping:

```
New in 1.3.

Alternative files can be specified for each inventory. One can give a tuple for the second inventory tuple item as shown in the 
following example. This will read the inventory iterating through the (second) tuple items until the first successful fetch. The
 primary use case for this to specify mirror sites for server downtime of the primary inventory:


intersphinx_mapping = {'python': ('https://docs.python.org/3',
                                  (None, 'python-inv.txt'))}
```

I think it is just a matter of:

a) automatically fetching the inventories from time to time
b) putting them on our S3 bucket with "web" property
c) adding automated mirroring from the S3 bucket.
d) we can even automatically update those S3 bucket inventories in our nightly scheduled build.

Should be easy to do and can save us a LOT of hassle.

Mirroring sounds like a good idea. What should happen if the nightly S3 bucket update fails?
> Mirroring sounds like a good idea. What should happen if the nightly S3 bucket update fails?

I think we somehow need to ignore the inventories if possible or building those providers if inventories are absolutely required. Would you like to take a stab at this @uranusjr :) ?
Sure! Could you provide some pointers where I should start looking for the code/configuration for the nightly builds?
I think I’ve it mostly figured out. Since we are already downloading the inventory locally ourselves into `docs/_inventory_cache`, we don’t really need to use the tuple form mentioned above. Instead, we can leave `conf.py` unchanged, but instead change `fetch_inventories.py` to check the mirror if the primary source fails during `_inventory_cache` population.

As mentioned above, the mirror inventory can be populated periodically. So a can will be added to `ci.yml` to fetch the inventory and populate them to S3.

Does that sound like a plan? I’ll try to come up with something.
First stab in #15024.
I think the build job in CI should always use the S3 cache - it is much more predictable and under our control that way.

And if we have the mirror as a separate workflow, then it can be triggered manually if we want to update it sooner than the nightly job.
Looks great! @uranusjr !

I have another thought in the meantime. Why not using caching of GitHub Actions: https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows - the inventory files are not big (2.3 M):

```
[jarek:~/code/airflow/docs] master+ ± du -h --summarize _inventory_cache
2.3M	_inventory_cache
```

So we could simply cache it and use already existing inventory cache mechanism this way (we are already using the _inventory_cache as fallback)?  That will probably be way simpler than setting up S3 and managing it periodically?

Also we could improve it to automatically add  the "exact" version of each library we use in each inventory URL, that would make it much better - currently we are taking "latest" or "stable" but in fact this is wrong - we should look at our constraints file and use the exact version of the library we use - this should be rather simple to retrieve from constraints file. That would also make caching much more efficient - we could invalidate cache whenever any of the libraries change and get it rebuilt. And this works really nicely if we use `restore-keys:` part of caching already present in GithubAction.

Some pointers: 

* we could take current library versions from : https://raw.githubusercontent.com/apache/airflow/constraints-master/constraints-3.8.txt
* we could simply download the constraint file and it could be used to calculate cache validity:
          with this key: cache-inventory-${{ hashFiles('constrant-3.8.txt') }}
          and this restore-keys: cache-inventory-
          the restore-keys work in the way that if the hash of such constraint file changes, it will download the "latest available" cache matching the prefix and use it as a base (but then after it is rebuilt, it will upload a new version of cache with the changed hash for next job to use it.
* and we could also use the constraints derive the right URLs when building inventory URLs to download the inventory from
    
WDYT? Happy to help and brainstorm on it, but this would be much simpler operationally (no separate S3 folder, no access needed etc. ) and more "correct" in terms of the documentation generated.
    
    
   
Using GitHub Action’s cache sounds like a cool idea. The only concern I have is the cache can be pretty difficult to purge if something goes wrong (maybe just me not knowing a good way to do it). Maybe we should have a “cache busting” part to the key (e.g. `cache-inventory-${{ hashFiles('constrant-3.8.txt') }}-${{ cacheKey }}` where `cacheKey` is a variable in the Action’s config) so we can force a cache miss when we need to.

Generally constraints (and requirements) file syntax is considered pip internals, but the files in this case are entirely managed by Airflow anyway, so that’s not really a concern.

We’ll probably need check if each third-party has a versioned documentation as well (not a given). And if one doesn’t, just default to `latest`?
I just thought of another thing, does Action’s cache set file creation date etc. correctly? Because if it does not, we won’t be able to fetch in new documentation updates (say the `1.4` version of a documentation releases a fix without changing the URL) since the files we retrieve from the cache will always be newer.
> Using GitHub Action’s cache sounds like a cool idea. The only concern I have is the cache can be pretty difficult to purge if something goes wrong (maybe just me not knowing a good way to do it). Maybe we should have a “cache busting” part to the key (e.g. cache-inventory-${{ hashFiles('constrant-3.8.txt') }}-${{ cacheKey }} where cacheKey is a variable in the Action’s config) so we can force a cache miss when we need to.

In this case we will simply change the cache prefix. We've done that several times with other caches. We can always turn it into:

```
key: cache-inventory-v2-${{ hashFiles('constraints-3.8.txt') }}
restore-key: cache-inventory-v2-
```

by changing the .yml file. No need for special variable for that it can simply be changed in ci.yml. And I think this cache will be self-healing. As I understand it the inventory we are using now simply downloads URL and only if it fails, it reads it from the cache @mik-laj might confirm it maybe?

> I just thought of another thing, does Action’s cache set file creation date etc. correctly? Because if it does not, we won’t be able to fetch in new documentation updates (say the `1.4` version of a documentation releases a fix without changing the URL) since the files we retrieve from the cache will always be newer.

Highly doubt it. Even if they do, this is not a big problem for us. First of all the cache will always be secondary choice - so even if we have it from the cache, the "URL one" will take precedence (Or this is at least I think how it works ). And secondly the cache will get "eventually consistent". Every time we change constraints (should happen regularly - every day or so) the cache will get refreshed because while we will have it as secondary choice, the main one (URL pull) will still get the latest version and override and then new version of the cache will be produced.
> We’ll probably need check if each third-party has a versioned documentation as well (not a given). And if one doesn’t, just default to latest?

Yeah. I think this is part of the solution, but I guess it will be easy by just running the inventory pull and seeing when it fails  (and make some exception list).
> As I understand it the inventory we are using now simply downloads URL and only if it fails, it reads it from the cache @mik-laj might confirm it maybe?

Not exactly. Before building, we download all inventories to the cache. If it is a third -party library, we download the inventory from the project documentation.  If this is our package then we download it from the S3 bucket -http://s.apache.org/airflow-docs
If documentation for a package has already been built, the locally built documentation inventory is used, not the cache i.e. prefer newer inwentories. 

The first package uses only use cached data. There should never be new inventories from the internet when building.
> If it is a third -party library, we download the inventory from the project documentation

So I guess the solution is to : 

1) Restore cache always from github's cache with restore-key: fallback when hash of constraints changed
2) Try to download from 3rd-party to that cache (and if we can't load we just skip the download rather than fail the build)
3) Build the docs from cache

Then if the fallback cache has been chosen the new cache will be created using the newly updated cache

We can even filter out the the constraint file to only contain the 'relevant' libraries and base our hash on that filtered file. Then the cache will only get updated when we update to a newer version of relevant libraries only.

That looks pretty sound to me @uranusjr @mik-laj  WDYT ?


We should make sure that the documentation for our packages is also built with the cache from the S3 bucket. In some cases, the Github Actiion cache may be empty, which will make our build take a very long time and it is very likely that we will have a timeout.  We can skip retrieving inventories from the bucket if we can restore the files from Github Acttion.

Resolving patch:
diff --git a/docs/exts/docs_build/fetch_inventories.py b/docs/exts/docs_build/fetch_inventories.py
--- a/docs/exts/docs_build/fetch_inventories.py
+++ b/docs/exts/docs_build/fetch_inventories.py
@@ -20,10 +20,13 @@
 import datetime
 import os
 import shutil
+import sys
+import traceback
 from itertools import repeat
 from typing import Iterator, List, Tuple
 
 import requests
+import urllib3.exceptions
 from requests.adapters import DEFAULT_POOLSIZE
 
 from airflow.utils.helpers import partition
@@ -50,9 +53,15 @@ def _fetch_file(session: requests.Session, package_name: str, url: str, path: st
     Download a file and returns status information as a tuple with package
     name and success status(bool value).
     """
-    response = session.get(url, allow_redirects=True, stream=True)
+    try:
+        response = session.get(url, allow_redirects=True, stream=True)
+    except (requests.RequestException, urllib3.exceptions.HTTPError):
+        print(f"Failed to fetch inventory: {url}")
+        traceback.print_exc(file=sys.stderr)
+        return package_name, False
     if not response.ok:
         print(f"Failed to fetch inventory: {url}")
+        print(f"Failed with status: {response.status_code}", file=sys.stderr)
         return package_name, False
 
     os.makedirs(os.path.dirname(path), exist_ok=True)


Test patch: