Repository: easybuilders/easybuild-easyblocks

Issue:
Bundle easyblock and downloading of files.
I ran into an issue with the Bundle EasyBlock that could either be a feature with undesirable side-effects or a bug depending on how you look at it. See the EasyConfig which I attached to this message.

It turns out that Bundle during the fetch phase throws the "source_urls" for all components into one big pool, then starts fetching the requested files not trying only the source_urls given for that bundle component, but trying all given source_urls in order. In the attached EasyConfig this goes wrong after the second bundle component (libcerf). The server serving that source_urls has the undesirable feature to not produce an error message when requesting a different .tar.gz or .tar.bz2 file than the one served (libcerf) and instead create a file with the requested name and the latest version of libcerf into it. To see this (apart from the text in the log files), go to the directory where sources are stored after the fetch step and do a tar -tf of gmp-6.1.2.tar.bz2, mpc-1.1.0.tar.gz and mpfr-4.0.2.tar.xz. The file for mpfr has the expected contents, the once for gmp and mpc do not but contains a version of libcerf.

It is not an absolute showstopper. In fact, in comments in the EasyConfig I show the workaround (by specifying the source location of libcerf in a different way), but it is very confusing while developing the bundle EasyConfig... Of course one could blame the server, but I've run in other servers already with similar undesirable behaviour when installing packages that have multiple source files (I don't remember which one), where one needed to be very careful with the order of the URLs (which again I solved by using the same trick as I did in this EasyConfig to make things more robust).

Attachement: a sample EasyConfig showing the problem.
[baselibs-devel-2019a-GCCcore-8.2.0-downloadMPC.eb.txt](https://github.com/easybuilders/easybuild-easyblocks/files/3043435/baselibs-devel-2019a-GCCcore-8.2.0-downloadMPC.eb.txt)



Discussion:
@klust What version of EB are you using? I would've thought from reading https://github.com/easybuilders/easybuild-easyblocks/blob/master/easybuild/easyblocks/generic/bundle.py#L109 that the `source_urls` are reset for each component in the bundle?
I tested your easyconfig with `--inject-checksums` and it worked correctly (different checksums and correct contents).
Do you download your sources in a particular way? I did notice that when using `-x` there didn't appear to be any distinction and I wonder if `--fetch` might throw a problem?
Sorry, I repeated that and I can reproduce your issue now. I'll see if I can figure out what is going on.
I'm thinking about how to fix this. The issue arises at https://github.com/easybuilders/easybuild-easyblocks/blob/master/easybuild/easyblocks/generic/bundle.py#L128 which is indeed just appending each `source_urls` to a global list (from what I can tell). I think it might be hard to change this behaviour (at least for me).

What I was thinking of doing was introducing the capability to check if a URL throws an EB-catchable failure when we give it a random file. If it doesn't we can tolerate one of these failures (by placing that URL at the end of the list) but if there are two such failures we would need to error out (unless a checksum has been provided, which would allow people to manually solve the problem). Thoughts? (@boegel )
The other option is to override the `fetch_step`...that looks complicated.
Ok, I think I have a solution, we treat the bundle components the same as we do extensions (which are a special case in the `fetch_step`). I'll give it a try.
There's really two problems here:

1. The download mechanism not detecting that what is being downloaded is not what is intended.

   We should enhance the download mechanism to at the very least check whether the file being downloaded has the correct type (i.e. is not an HTML page rather than a gzipped tarball). But that may not fix the problem in this case, of course.

   Of course, using SHA256 checksums do help in this case, since the checksum check will fail if something else gets downloaded than what is intended (at least, if the checksum for the right file was put in place). So it may not have avoided the confusing situation that @klust found himself in (`--inject-checksums` definitely doesn't help here).

2. The fact that `Bundle` builds up a (potentially very long) list of `source_urls` by combining the ones from each component gradually is definitely a bug...

   I think the intention of the current code is to combine the component-specific `source_urls` (if any) with the global `source_urls` specified in the easyconfig file (if any, cfr. `X11` easyconfigs).

   Of course, that may goes south for the case shown by @klust.

   The solution for this, I think, is to make `fetch_step` aware of the components, and only try the global `source_urls` (which empty in the case of @klust's example) + the component-specific `source_urls`, and avoid that the per component `source_urls` are inherited across components.
   That probably also involves not building a huge list of `sources` & `checksums` too (which is just a hacky way of making `fetch_step` pull in everything for all components).
This is being addressed in https://github.com/easybuilders/easybuild-framework/pull/2855 and #1710 but it is actually quite complicated to resolve. We need to elevate components to the same status as extensions in the framework.
@ocaisa Sorry for not answering sooner, I was at a course last week. I don't think the answer still matters, but given that I filed on April 4 I must have been using EasyBuild 3.8.1. On our site we tend to always be using the latest released version within days after release. I may have checked against the development version also, I had that one set up properly already back then.

@boegel 2 (combining the global source_urls with the component-specific ones, and then prefereable the component-specific ones first) would at least have been the expected behaviour. In fact, in this case it would have helped me. Though I agree that point 1 you raise is indeed a problem also. I've had other non-bundle easyconfigs where I carefully needed to order the URLs and downloaded files as otherwise I'd hit a repository with "bad" behaviour returning a HTML file rather than just an error code.

Resolving patch:
diff --git a/easybuild/easyblocks/generic/bundle.py b/easybuild/easyblocks/generic/bundle.py
--- a/easybuild/easyblocks/generic/bundle.py
+++ b/easybuild/easyblocks/generic/bundle.py
@@ -85,60 +85,72 @@ def __init__(self, *args, **kwargs):
             if len(comp) == 3:
                 comp_specs = comp[2]
 
-            cfg = self.cfg.copy()
+            comp_cfg = self.cfg.copy()
 
             easyblock = comp_specs.get('easyblock') or self.cfg['default_easyblock']
             if easyblock is None:
-                raise EasyBuildError("No easyblock specified for component %s v%s", cfg['name'], cfg['version'])
+                raise EasyBuildError("No easyblock specified for component %s v%s", comp_cfg['name'],
+                                     comp_cfg['version'])
             elif easyblock == 'Bundle':
                 raise EasyBuildError("The Bundle easyblock can not be used to install components in a bundle")
 
-            cfg.easyblock = get_easyblock_class(easyblock, name=cfg['name'])
+            comp_cfg.easyblock = get_easyblock_class(easyblock, name=comp_cfg['name'])
 
             # make sure that extra easyconfig parameters are known, so they can be set
-            extra_opts = cfg.easyblock.extra_options()
-            cfg.extend_params(copy.deepcopy(extra_opts))
+            extra_opts = comp_cfg.easyblock.extra_options()
+            comp_cfg.extend_params(copy.deepcopy(extra_opts))
 
-            cfg['name'] = comp_name
-            cfg['version'] = comp_version
-            cfg.generate_template_values()
+            comp_cfg['name'] = comp_name
+            comp_cfg['version'] = comp_version
+            comp_cfg.generate_template_values()
 
             # do not inherit easyblock to use from parent (since that would result in an infinite loop in install_step)
-            cfg['easyblock'] = None
+            comp_cfg['easyblock'] = None
 
             # reset list of sources/source_urls/checksums
-            cfg['sources'] = cfg['source_urls'] = cfg['checksums'] = []
+            comp_cfg['sources'] = comp_cfg['source_urls'] = comp_cfg['checksums'] = []
 
             for key in self.cfg['default_component_specs']:
-                cfg[key] = self.cfg['default_component_specs'][key]
+                comp_cfg[key] = self.cfg['default_component_specs'][key]
 
             for key in comp_specs:
-                cfg[key] = comp_specs[key]
+                comp_cfg[key] = comp_specs[key]
 
             # enable resolving of templates for component-specific EasyConfig instance
-            cfg.enable_templating = True
+            comp_cfg.enable_templating = True
 
             # 'sources' is strictly required
-            if cfg['sources']:
-                # add component sources to list of sources
-                self.cfg.update('sources', cfg['sources'])
+            if comp_cfg['sources']:
+                # If per-component source URLs are provided, attach them directly to the relevant sources
+                if comp_cfg['source_urls']:
+                    for source in comp_cfg['sources']:
+                        if isinstance(source, basestring):
+                            self.cfg.update('sources', [{'filename': source, 'source_urls': comp_cfg['source_urls']}])
+                        elif isinstance(source, dict):
+                            # Update source_urls in the 'source' dict to use the one for the components
+                            # (if it doesn't already exist)
+                            if 'source_urls' not in source:
+                                source['source_urls'] = comp_cfg['source_urls']
+                            self.cfg.update('sources', [source])
+                        else:
+                            raise EasyBuildError("Source %s for component %s is neither a string nor a dict, cannot "
+                                                 "process it.", source, comp_cfg['name'])
+                else:
+                    # add component sources to list of sources
+                    self.cfg.update('sources', comp_cfg['sources'])
             else:
                 raise EasyBuildError("No sources specification for component %s v%s", comp_name, comp_version)
 
-            if cfg['source_urls']:
-                # add per-component source_urls to list of bundle source_urls, expanding templates
-                self.cfg.update('source_urls', cfg['source_urls'])
-
-            if cfg['checksums']:
-                src_cnt = len(cfg['sources'])
+            if comp_cfg['checksums']:
+                src_cnt = len(comp_cfg['sources'])
 
                 # add per-component checksums for sources to list of checksums
-                self.cfg.update('checksums', cfg['checksums'][:src_cnt])
+                self.cfg.update('checksums', comp_cfg['checksums'][:src_cnt])
 
                 # add per-component checksums for patches to list of checksums for patches
-                checksums_patches.extend(cfg['checksums'][src_cnt:])
+                checksums_patches.extend(comp_cfg['checksums'][src_cnt:])
 
-            self.comp_cfgs.append(cfg)
+            self.comp_cfgs.append(comp_cfg)
 
         self.cfg.update('checksums', checksums_patches)
 


Test patch: