Repository: mesonbuild/meson

Issue:
Make Meson recover from bad download
**Describe the bug**
When meson encounters an error while downloading a dependency, it might get into inconsistent state with following error:
```
meson.build:147:0: ERROR: Subproject exists but has no meson.build file
```

**To Reproduce**
```meson
# meson.build
project('test', 'c')

executable('main', 'main.c',
  dependencies: [
    subproject('libuv').get_variable('libuv_dep')
  ]
)
```
```c
// main.c
int main(int argc, char **argv) {
    return 0;
}
```
```ini
# subprojects/libuv.wrap
[wrap-file]
directory = libuv-v1.42.0
source_url = https://dist.libuv.org/dist/v1.42.0/libuv-v1.42.0.tar.gz
source_filename = libuv-v1.42.0.tar.gz
source_hash = 43129625155a8aed796ebe90b8d4c990a73985ec717de2b2d5d3a23cfe4deb72
patch_filename = libuv_1.42.0-1_patch.zip
patch_url = https://bad-wrapdb.mesonbuild.com/v2/libuv_1.42.0-1/get_patch
patch_hash = 891698d781bc68379b799e47bc7e100698f03985b6db5d1ddc4eaa9be42a8f35

[provide]
libuv = libuv_dep
```

`bad-wrapdb` above will simulate a network error (in practice error might be caused by network timeout, temporary DNS resolution issue, etc.).

Now do usual `meson setup build` and you'll get an obvious error.

If you replace `bad-wrapdb` with `wrapdb` to simulate the end of intermittent network issues you'll get:
```
meson.build:3:0: ERROR: Subproject exists but has no meson.build file
```

**Expected behavior**
Meson shoudn't keep subprojects in inconsistent state when such issues happen and thus should be able to recover without requiring user to go and manually remove directories under `subprojects`.

Retries would also be nice to have I think, like 3 attempts with exponential backoff.

**system parameters**
* Plain native build
* Ubuntu 21.04
* Python 3.9.5
* Meson 0.60.2
* Ninja 1.10.2.git.kitware.jobserver-1



Discussion:
You can recover that with `meson subprojects update --reset`.

I guess deleting it makes sense, for cases where we know we just created it and therefore it cannot have pre-existing user content, potentially modified.
Yes, I'm just concerned with simple cases like described where Meson knows that things will definitely break.
We have a bit of funding available for this issue at mediasoup thanks to the donations project receives.

Requirements:
* [ ] make meson remove incomplete subproject files in case of network error to avoid "Subproject exists but has no meson.build file" error as described above
* [ ] add support for retries in case network connection has issues (3-5 attempts with exponential backoff would be ideal)
* [ ] submit as PR to meson
* [ ] get PR merged by meson maintainers

After that go to https://opencollective.com/mediasoup and send **$100 USD** expense to get paid for your work.

Resolving patch:
diff --git a/mesonbuild/wrap/wrap.py b/mesonbuild/wrap/wrap.py
--- a/mesonbuild/wrap/wrap.py
+++ b/mesonbuild/wrap/wrap.py
@@ -26,13 +26,14 @@
 import subprocess
 import sys
 import configparser
+import time
 import typing as T
 import textwrap
 
 from pathlib import Path
 from . import WrapMode
 from .. import coredata
-from ..mesonlib import quiet_git, GIT, ProgressBar, MesonException
+from ..mesonlib import quiet_git, GIT, ProgressBar, MesonException, windows_proof_rmtree
 from ..interpreterbase import FeatureNew
 from .. import mesonlib
 
@@ -345,7 +346,11 @@ def resolve(self, packagename: str, method: str) -> str:
                     self.get_svn()
                 else:
                     raise WrapException(f'Unknown wrap type {self.wrap.type!r}')
-            self.apply_patch()
+            try:
+                self.apply_patch()
+            except:
+                windows_proof_rmtree(self.dirname)
+                raise
 
         # A meson.build or CMakeLists.txt file is required in the directory
         if method == 'meson' and not os.path.exists(meson_file):
@@ -530,12 +535,22 @@ def check_hash(self, what: str, path: str, hash_required: bool = True) -> None:
         if dhash != expected:
             raise WrapException(f'Incorrect hash for {what}:\n {expected} expected\n {dhash} actual.')
 
+    def get_data_with_backoff(self, urlstring: str) -> T.Tuple[str, str]:
+        delays = [1, 2, 4, 8, 16]
+        for d in delays:
+            try:
+                return self.get_data(urlstring)
+            except Exception as e:
+                mlog.warning(f'failed to download with error: {e}. Trying after a delay...', fatal=False)
+                time.sleep(d)
+        return self.get_data(urlstring)
+
     def download(self, what: str, ofname: str, fallback: bool = False) -> None:
         self.check_can_download()
         srcurl = self.wrap.get(what + ('_fallback_url' if fallback else '_url'))
         mlog.log('Downloading', mlog.bold(self.packagename), what, 'from', mlog.bold(srcurl))
         try:
-            dhash, tmpfile = self.get_data(srcurl)
+            dhash, tmpfile = self.get_data_with_backoff(srcurl)
             expected = self.wrap.get(what + '_hash').lower()
             if dhash != expected:
                 os.remove(tmpfile)


Test patch: