Repository: nf-core/tools

Issue:
`create`: check HTML response is 200 when fetching logo
### Description of the bug

After an automated sync of the nf-core tools v2.2, the dark logo for Epitopeprediction was not created and instead this was inside that file:

```html 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>507 Insufficient Storage</title>
</head><body>
<h1>Insufficient Storage</h1>
<p>The method could not be performed on the resource
because the server is unable to store the
representation needed to successfully complete the
request.  There is insufficient free space left in
your storage allocation.</p>
<p>Additionally, a 507 Insufficient Storage
error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body></html>
```

I obtained the same results when running `nf-core sync` locally.

This also causes an nf-core lint error as the file "is not equal to the template"


### Command used and terminal output

```console
nf-core sync
```


### System information

I'm not sure what are the versions installed in the server.

Locally I was using `NXF_VER = 21.10.5` and nf-core tools `v2.2` on macOS
`create`: check HTML response is 200 when fetching logo
### Description of the bug

After an automated sync of the nf-core tools v2.2, the dark logo for Epitopeprediction was not created and instead this was inside that file:

```html 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>507 Insufficient Storage</title>
</head><body>
<h1>Insufficient Storage</h1>
<p>The method could not be performed on the resource
because the server is unable to store the
representation needed to successfully complete the
request.  There is insufficient free space left in
your storage allocation.</p>
<p>Additionally, a 507 Insufficient Storage
error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body></html>
```

I obtained the same results when running `nf-core sync` locally.

This also causes an nf-core lint error as the file "is not equal to the template"


### Command used and terminal output

```console
nf-core sync
```


### System information

I'm not sure what are the versions installed in the server.

Locally I was using `NXF_VER = 21.10.5` and nf-core tools `v2.2` on macOS


Discussion:
Okay, so you have it in the template, but it is not copied over correctly? Or why is the linting also complaining?
So the `nf-core create` command that is called by sync will be downloading that HTML instead of the dark logo from the nf-co.re website. I guess the code doesn't check whether we get a good response, so just saves whatever it gets to the file. Then linting fails because the file looks wrong..

I guess that this happens when the website is hit by all pipelines at once during the sync to generate a lot of images. It will have been especially severe here because we changed the filenames and added new logo, so instead of using the cached logo results, it will have needed to generate 100+ logos all at one time.

We could add some code to the `create` command to check a successful response, but as they should all be cached for future syncs I'm inclined to forget about it and it shouldn't happen again. If we change the logos again we should try to generate all of the logos manually prior to sync (I think I've done this before for the same reason actually).
Okay, sounds reasonable. How should we handle if the response is != 200? wait and try again?
In order to fix the sync related issue, should we add the logo creation as an explicit step?
Actually, I checked for cageseq and the image was correct in the assets directory (for the email template), but not in the docs. So, it doesn't look like a image creation / caching issue, more that it can't get everything delivered?
The issue was only for the dark logo, which I think was only in the bin folder
I think that the docs and assets images are different resolutions? So will be treated as totally separate -> still a generation error (needs cache).

I think it's fine to just wait a few seconds and retry if we get != 200 for the fetch, maybe with max 5 retries or something? Hopefully that should get most through. I feel like we did something similar already for creating PRs which failed when we had a surge..?
FWIW I got a separate error in the eager auto-sync

```html
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE> 508 Resource Limit Is Reached</TITLE>
</HEAD><BODY>
<H1>Resource Limit Is Reached</H1>
The website is temporarily unable to service your request as it exceeded resource limit.
Please try again later.
</BODY></HTML>
```

This was both in light and dark
Also if someone comes searching for how to fix it `nf-core lint --fix_files_unchanged` recreates the images correctly :+1: 
ah great to know @jfy133 🎉 
Okay, so you have it in the template, but it is not copied over correctly? Or why is the linting also complaining?
So the `nf-core create` command that is called by sync will be downloading that HTML instead of the dark logo from the nf-co.re website. I guess the code doesn't check whether we get a good response, so just saves whatever it gets to the file. Then linting fails because the file looks wrong..

I guess that this happens when the website is hit by all pipelines at once during the sync to generate a lot of images. It will have been especially severe here because we changed the filenames and added new logo, so instead of using the cached logo results, it will have needed to generate 100+ logos all at one time.

We could add some code to the `create` command to check a successful response, but as they should all be cached for future syncs I'm inclined to forget about it and it shouldn't happen again. If we change the logos again we should try to generate all of the logos manually prior to sync (I think I've done this before for the same reason actually).
Okay, sounds reasonable. How should we handle if the response is != 200? wait and try again?
In order to fix the sync related issue, should we add the logo creation as an explicit step?
Actually, I checked for cageseq and the image was correct in the assets directory (for the email template), but not in the docs. So, it doesn't look like a image creation / caching issue, more that it can't get everything delivered?
The issue was only for the dark logo, which I think was only in the bin folder
I think that the docs and assets images are different resolutions? So will be treated as totally separate -> still a generation error (needs cache).

I think it's fine to just wait a few seconds and retry if we get != 200 for the fetch, maybe with max 5 retries or something? Hopefully that should get most through. I feel like we did something similar already for creating PRs which failed when we had a surge..?
FWIW I got a separate error in the eager auto-sync

```html
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<HTML><HEAD>
<TITLE> 508 Resource Limit Is Reached</TITLE>
</HEAD><BODY>
<H1>Resource Limit Is Reached</H1>
The website is temporarily unable to service your request as it exceeded resource limit.
Please try again later.
</BODY></HTML>
```

This was both in light and dark
Also if someone comes searching for how to fix it `nf-core lint --fix_files_unchanged` recreates the images correctly :+1: 
ah great to know @jfy133 🎉 

Resolving patch:
diff --git a/nf_core/create.py b/nf_core/create.py
--- a/nf_core/create.py
+++ b/nf_core/create.py
@@ -4,13 +4,16 @@
 """
 from genericpath import exists
 import git
+import imghdr
 import jinja2
 import logging
 import os
 import pathlib
+import random
 import requests
 import shutil
 import sys
+import time
 
 import nf_core
 
@@ -149,18 +152,52 @@ def make_pipeline_logo(self):
         log.debug(f"Fetching logo from {logo_url}")
 
         email_logo_path = f"{self.outdir}/assets/{self.name_noslash}_logo_light.png"
-        os.makedirs(os.path.dirname(email_logo_path), exist_ok=True)
-        log.debug(f"Writing logo to '{email_logo_path}'")
-        r = requests.get(f"{logo_url}&w=400")
-        with open(email_logo_path, "wb") as fh:
-            fh.write(r.content)
+        self.download_pipeline_logo(f"{logo_url}&w=400", email_logo_path)
         for theme in ["dark", "light"]:
+            readme_logo_url = f"{logo_url}?w=600&theme={theme}"
             readme_logo_path = f"{self.outdir}/docs/images/{self.name_noslash}_logo_{theme}.png"
-            log.debug(f"Writing logo to '{readme_logo_path}'")
-            os.makedirs(os.path.dirname(readme_logo_path), exist_ok=True)
-            r = requests.get(f"{logo_url}?w=600&theme={theme}")
-            with open(readme_logo_path, "wb") as fh:
+            self.download_pipeline_logo(readme_logo_url, readme_logo_path)
+
+    def download_pipeline_logo(self, url, img_fn):
+        """Attempt to download a logo from the website. Retry if it fails."""
+        os.makedirs(os.path.dirname(img_fn), exist_ok=True)
+        attempt = 0
+        max_attempts = 10
+        retry_delay = 0  # x up to 10 each time, so first delay will be 1-100 seconds
+        while attempt < max_attempts:
+            # If retrying, wait a while
+            if retry_delay > 0:
+                log.info(f"Waiting {retry_delay} seconds before next image fetch attempt")
+                time.sleep(retry_delay)
+
+            attempt += 1
+            # Use a random number to avoid the template sync hitting the website simultaneously for all pipelines
+            retry_delay = random.randint(1, 100) * attempt
+            log.debug(f"Fetching logo '{img_fn}' (attempt {attempt})")
+            try:
+                # Try to fetch the logo from the website
+                r = requests.get(url, timeout=180)
+                if r.status_code != 200:
+                    raise UserWarning(f"Got status code {r.status_code}")
+                # Check that the returned image looks right
+
+            except (ConnectionError, UserWarning) as e:
+                # Something went wrong - try again
+                log.warning(e)
+                log.error(f"Connection error - retrying")
+                continue
+
+            # Write the new logo to the file
+            with open(img_fn, "wb") as fh:
                 fh.write(r.content)
+            # Check that the file looks valid
+            image_type = imghdr.what(img_fn)
+            if image_type != "png":
+                log.error(f"Logo from the website didn't look like an image: '{image_type}'")
+                continue
+
+            # Got this far, presumably it's good - break the retry loop
+            break
 
     def git_init_pipeline(self):
         """Initialises the new pipeline as a Git repository and submits first commit."""


Test patch: