Repository: easybuilders/easybuild-framework

Issue:
singularity build error issue with pip wheel and upgrade pip 
@boegel 

I was able to confirm that the generated singularity definition file is not working. 

I ran the following `eb Anaconda3-5.3.0.eb -C --container-config bootstrap=yum,osversion=7 --experimental` to build a definition file and then building this remotely caused an issue stating to upgrade pip and install `wheel` package. 

```
c455-051[knl](350)$ singularity build --remote Anaconda3-5.3.0.sif Singularity.Anaconda3-5.3.0
INFO:    Remote "default" added.
INFO:    Authenticating with remote: default
INFO:    API Key Verified!
INFO:    Remote "default" now in use.
INFO:    Starting build...
INFO:    Skipping GPG Key Import
INFO:    Running post scriptlet
+ yum install --quiet --assumeyes epel-release
warning: /var/cache/yum/x86_64/7/extras/packages/epel-release-7-11.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for epel-release-7-11.noarch.rpm is not installed
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-7.1908.0.el7.centos.x86_64 (@base)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
+ yum install --quiet --assumeyes python setuptools Lmod
http://csc.mcs.sdsmt.edu/epel/7/x86_64/repodata/549304b0255d4cedc8b8d012cbafdf28db891791aab41836de01758a2b511597-updateinfo.xml.bz2: [Errno 12] Timeout on http://csc.mcs.sdsmt.edu/epel/7/x86_64/repodata/549304b0255d4cedc8b8d012cbafdf28db891791aab41836de01758a2b511597-updateinfo.xml.bz2: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
Package python-2.7.5-86.el7.x86_64 already installed and latest version
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: Public key for Lmod-8.2.7-1.el7.x86_64.rpm is not installed
/var/cache/yum/x86_64/7/epel/packages/Lmod-8.2.7-1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Importing GPG key 0x352C64E5:
 Userid     : "Fedora EPEL (7) <epel@fedoraproject.org>"
 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
 Package    : epel-release-7-11.noarch (@extras)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Running in chroot, ignoring request.
+ yum install --quiet --assumeyes python-pip
+ yum install --quiet --assumeyes bzip2 gzip tar zip unzip xz
Package gzip-1.5-10.el7.x86_64 already installed and latest version
Package 2:tar-1.26-35.el7.x86_64 already installed and latest version
Package xz-5.2.2-1.el7.x86_64 already installed and latest version
+ yum install --quiet --assumeyes curl wget
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
+ yum install --quiet --assumeyes patch make
+ yum install --quiet --assumeyes file git which
+ yum install --quiet --assumeyes gcc-c++
+ yum install --quiet --assumeyes perl-Data-Dumper
+ yum install --quiet --assumeyes perl-Thread-Queue
+ yum --skip-broken --quiet --assumeyes install libibverbs-dev libibverbs-devel rdma-core-devel
+ yum --skip-broken --quiet --assumeyes install openssl-devel libssl-dev libopenssl-devel
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
+ pip install -U setuptools
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/54/28/c45d8b54c1339f9644b87663945e54a8503cfef59cf0f65b3ff5dd17cf64/setuptools-42.0.2-py2.py3-none-any.whl (583kB)
Installing collected packages: setuptools
  Found existing installation: setuptools 0.9.8
    Uninstalling setuptools-0.9.8:
      Successfully uninstalled setuptools-0.9.8
Successfully installed setuptools-42.0.2
You are using pip version 8.1.2, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
+ pip install 'vsc-install<0.11.4' 'vsc-base<2.9.0'
Collecting vsc-install<0.11.4
  Downloading https://files.pythonhosted.org/packages/18/59/3274a58af6af84a87f7655735b452c06c769586ee73954f5ee15d303aa29/vsc-install-0.11.3.tar.gz (61kB)
Collecting vsc-base<2.9.0
  Downloading https://files.pythonhosted.org/packages/48/aa/f05d350c358338d0e843835660e3993cc5eb28401f32c0c5b8bc9a9458d5/vsc-base-2.8.4.tar.gz (104kB)
    Complete output from command python setup.py egg_info:
    INFO: This is (based on) vsc.install.shared_setup 0.11.5
    INFO: run_tests from base dir /tmp/pip-build-VJzWRr/vsc-base (using executable /tmp/pip-build-VJzWRr/vsc-base/-c)
    WARN: cleanup lib/vsc_base.egg-info
    INFO: initial packages list: ['vsc', 'vsc.utils']
    INFO: generated list: ['vsc', 'vsc.utils']
    INFO: generated packages list: ['vsc', 'vsc.utils']
    INFO: makesetupcfg set to True, (re)creating setup.cfg
    INFO: found license /tmp/pip-build-VJzWRr/vsc-base/LICENSE with md5sum 5f30f0716dfdd0d91eb439ebec522ec2
    INFO: Found license name LGPLv2+ and classifier License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)
    INFO: setting license LGPLv2+
    INFO: found match url https://github.com/hpcugent/vsc-base in /tmp/pip-build-VJzWRr/vsc-base/PKG-INFO
    INFO: found match name vsc-base in /tmp/pip-build-VJzWRr/vsc-base/PKG-INFO
    INFO: Removing None download_url
    INFO: get_name_url returns {'url': 'https://github.com/hpcugent/vsc-base', 'name': 'vsc-base'}
    INFO: using long_description Common tools used within our organization. Originally created by the HPC team of Ghent University (http://ugent.be/hpc).
    INFO: generated list: ['bin/logdaemon.py', 'bin/optcomplete.bash', 'bin/startlogdaemon.sh']
    INFO: generated scripts list: ['bin/logdaemon.py', 'bin/optcomplete.bash', 'bin/startlogdaemon.sh']
    INFO: adding prospector to tests_require
    {'maintainer': 'Stijn De Weirdt;Jens Timmerman;Andy Georges;Kenneth Hoste', 'install_requires': ['vsc-install >= 0.10.2'], 'setup_requires': ['vsc-install >= 0.10.2'], 'maintainer_email': 'stijn.deweirdt@ugent.be, jens.timmerman@ugent.be, andy.georges@ugent.be, kenneth.hoste@ugent.be', 'long_description': "# vsc-base\n\n### Build Status\n\n- Python 2.6 : [![Build Status](https://jenkins1.ugent.be/job/vsc-base-python26/badge/icon)](https://jenkins1.ugent.be/job/vsc-base-python26/)\n- Python 2.7 : [![Build Status](https://jenkins1.ugent.be/job/vsc-base-python27/badge/icon)](https://jenkins1.ugent.be/job/vsc-base-python27/)\n\n# Description\n\nCommon tools used within our organization.\nOriginally created by the HPC team of Ghent University (http://ugent.be/hpc).\n\n# Documentation\nhttps://jenkins1.ugent.be/job/vsc-base-python26/Documentation/\n\n# Namespaces and tools\n\n## lib/utils\npython utilities to be used as libraries\n\n- __fancylogger__: an extention of the default python logger designed to be easy to use and have a couple of `fancy` features.\n\n - custom specifiers for mpi loggin (the mpirank) with autodetection of mpi\n - custom specifier for always showing the calling function's name\n - rotating file handler\n - a default formatter.\n - logging to an UDP server (logdaemon.py f.ex.)\n - easily setting loglevel\n\n- __daemon.py__ : Daemon class written by Sander Marechal (http://www.jejik.com) to start a python script as a daemon.\n- __missing.py__: Small functions and tools that are commonly used but not available in the Python (2.x) API.\n- ~~__cache.py__ : File cache to store pickled data identified by a key accompanied by a timestamp.~~ (moved to [vsc-utils](https://github.com/hpcugent/vsc-utils))\n- __generaloption.py__ : A general option parser for python. It will fetch options (in this order) from config files, from environment variables and from the command line and parse them in a way compatible with the default python optionparser. Thus allowing a very flexible way to configure your scripts. It also adds a few other useful extras.\n- __affinity.py__ : Linux cpu affinity.\n\n - Based on `sched.h` and `bits/sched.h`,\n - see man pages for `sched_getaffinity` and `sched_setaffinity`\n - also provides a `cpuset` class to convert between human readable cpusets and the bit version Linux priority\n - Based on sys/resources.h and bits/resources.h see man pages for `getpriority` and `setpriority`\n\n- __asyncprocess.py__ : Module to allow Asynchronous subprocess use on Windows and Posix platforms\n\n - Based on a [python recipe](http://code.activestate.com/recipes/440554/) by Josiah Carlson\n - added STDOUT handle and recv_some\n\n- __daemon.py__ : [A generic daemon class by Sander Marechal](http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/)\n- __dateandtime.py__ : A module with various convenience functions and classes to deal with date, time and timezone.\n- __nagios.py__ : This module provides functionality to cache and report results of script executions that can readily be interpreted by nagios/icinga.\n- __run.py__ : Python module to execute a command, can make use of asyncprocess, answer questions based on a dictionary\n\n - supports a whole lot of ways to input, process and output the command. (filehandles, PIPE, pty, stdout, logging...)\n\n- __mail.py__ : Wrapper around the standard Python mail library.\n\n - Send a plain text message\n - Send an HTML message, with a plain text alternative\n\n## bin\nA collection of python scripts, these are examples of how you could use fancylogger to log to a daemon, but should not be used directly.\n- __logdaemon.py__: A daemon that listens on a port for udp packets and logs them to file, works toghether with fancylogger.\n- __startlogdaemon.py__ : Script that will start the logdaemon for  you and set environment variables for fancylogger.\n\n# License\nvsc-base is made available under the GNU Library General Public License (LGPL) version 2 or any later version.\n\n# Acknowledgements\nvsc-base was created with support of [Ghent UniversityWARNING: The wheel package is not available.
    ERROR: 'pip wheel' requires the 'wheel' package. To fix this, run: pip install wheel
    ](http://www.ugent.be/en),\nthe [Flemish Supercomputer Centre (VSC)](https://vscentrum.be/nl/en),\nthe [Flemish Research Foundation (FWO)](http://www.fwo.be/en),\nand [the Department of Economy, Science and Innovation (EWI)](http://www.ewi-vlaanderen.be/en).\n\n", 'tests_require': ['prospector', 'coloredlogs<6.0', 'humanfriendly', 'prospector >= 1.1.4'], 'package_dir': {'': 'lib'}, 'extras_require': {'coloredlogs': ['coloredlogs<6.0', 'humanfriendly']}, 'author': 'Stijn De Weirdt;Jens Timmerman;Andy Georges;Kenneth Hoste', 'cmdclass': {'bdist_rpm': <class shared_setup_dist_only.vsc_bdist_rpm at 0x7fea974da2c0>, 'install_scripts': <class shared_setup_dist_only.vsc_install_scripts at 0x7fea974da390>, 'test': <class shared_setup_dist_only.VscTestCommand at 0x7fea974da1f0>, 'vsc_release': <class shared_setup_dist_only.vsc_release at 0x7fea974da120>, 'egg_info': <class shared_setup_dist_only.vsc_egg_info at 0x7fea9753d738>, 'sdist': <class shared_setup_dist_only.vsc_sdist at 0x7fea9753dc18>}, 'download_url': '', 'version': '2.8.4', 'description': 'Common tools used within our organization. Originally created by the HPC team of Ghent University (http://ugent.be/hpc).', 'namespace_packages': ['vsc'], 'scripts': ['bin/logdaemon.py', 'bin/optcomplete.bash', 'bin/startlogdaemon.sh'], 'command_packages': ['vsc.install.shared_setup', 'shared_setup_dist_only', 'setuptools.command', 'distutils.command'], 'packages': ['vsc', 'vsc.utils'], 'classifiers': ['License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)'], 'dependency_links': ['git+https://github.com/hpcugent/vsc-install.git#egg=vsc-install-0.10.2', 'git+https://github.com/hpcugent/vsc-install.git#egg=vsc-install-0.10.2'], 'name': 'vsc-base', 'license': 'LGPLv2+', 'url': 'https://github.com/hpcugent/vsc-base', 'author_email': 'stijn.deweirdt@ugent.be, jens.timmerman@ugent.be, andy.georges@ugent.be, kenneth.hoste@ugent.be', 'test_suite': 'test'}
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-VJzWRr/vsc-base/setup.py", line 68, in <module>
        shared_setup.action_target(PACKAGE)
      File "/tmp/pip-build-VJzWRr/vsc-base/external_dist_only/shared_setup_dist_only.py", line 1519, in action_target
        _fvs('action_target function')().action_target(package, *args, **kwargs)
      File "/tmp/pip-build-VJzWRr/vsc-base/external_dist_only/shared_setup_dist_only.py", line 1507, in action_target
        setupfn(**x)
      File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 144, in setup
        _install_setup_requires(attrs)
      File "/usr/lib/python2.7/site-packages/setuptools/__init__.py", line 139, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 721, in fetch_build_eggs
        replace_conflicting=True,
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 782, in resolve
        replace_conflicting=replace_conflicting
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1065, in best_match
        return self.obtain(req, installer)
      File "/usr/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1077, in obtain
        return installer(requirement)
      File "/usr/lib/python2.7/site-packages/setuptools/dist.py", line 777, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "/usr/lib/python2.7/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
        raise DistutilsError(str(e))
    distutils.errors.DistutilsError: Command '['/usr/bin/python2', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpkCk47d', '--quiet', '--find-links', 'git+https://github.com/hpcugent/vsc-install.git#egg=vsc-install-0.10.2', '--find-links', 'git+https://github.com/hpcugent/vsc-install.git#egg=vsc-install-0.10.2', 'vsc-install>=0.10.2']' returned non-zero exit status 1

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-VJzWRr/vsc-base/
You are using pip version 8.1.2, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
FATAL:   failed to execute %post proc: exit status 1
FATAL:   While performing build: while running engine: while running /usr/local/libexec/singularity/bin/starter: exit status 255
FATAL:   While performing build: build image size <= 0

```

By adding the following two lines in the pip section fixed the problem

```
# install EasyBuild using pip
pip install --upgrade pip
pip install wheel
```

```
c455-002[knl](340)$ singularity build --remote Anaconda3-5.3.0.sif Singularity.Anaconda3-5.3.0
INFO:    Remote "default" added.
INFO:    Authenticating with remote: default
INFO:    API Key Verified!
INFO:    Remote "default" now in use.
INFO:    Starting build...
INFO:    Skipping GPG Key Import
INFO:    Running post scriptlet
+ yum install --quiet --assumeyes epel-release
warning: /var/cache/yum/x86_64/7/extras/packages/epel-release-7-11.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for epel-release-7-11.noarch.rpm is not installed
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-7.1908.0.el7.centos.x86_64 (@base)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
+ yum install --quiet --assumeyes python setuptools Lmod
http://csc.mcs.sdsmt.edu/epel/7/x86_64/repodata/549304b0255d4cedc8b8d012cbafdf28db891791aab41836de01758a2b511597-updateinfo.xml.bz2: [Errno 12] Timeout on http://csc.mcs.sdsmt.edu/epel/7/x86_64/repodata/549304b0255d4cedc8b8d012cbafdf28db891791aab41836de01758a2b511597-updateinfo.xml.bz2: (28, 'Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds')
Trying other mirror.
Package python-2.7.5-86.el7.x86_64 already installed and latest version
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: Public key for Lmod-8.2.7-1.el7.x86_64.rpm is not installed
/var/cache/yum/x86_64/7/epel/packages/Lmod-8.2.7-1.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Importing GPG key 0x352C64E5:
 Userid     : "Fedora EPEL (7) <epel@fedoraproject.org>"
 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
 Package    : epel-release-7-11.noarch (@extras)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Running in chroot, ignoring request.
+ yum install --quiet --assumeyes python-pip
+ yum install --quiet --assumeyes bzip2 gzip tar zip unzip xz
Package gzip-1.5-10.el7.x86_64 already installed and latest version
Package 2:tar-1.26-35.el7.x86_64 already installed and latest version
Package xz-5.2.2-1.el7.x86_64 already installed and latest version
+ yum install --quiet --assumeyes curl wget
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
+ yum install --quiet --assumeyes patch make
+ yum install --quiet --assumeyes file git which
+ yum install --quiet --assumeyes gcc-c++
+ yum install --quiet --assumeyes perl-Data-Dumper
+ yum install --quiet --assumeyes perl-Thread-Queue
+ yum --skip-broken --quiet --assumeyes install libibverbs-dev libibverbs-devel rdma-core-devel
+ yum --skip-broken --quiet --assumeyes install openssl-devel libssl-dev libopenssl-devel
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
+ pip install --upgrade pip
Collecting pip
  Downloading https://files.pythonhosted.org/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl (1.4MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.2
    Uninstalling pip-8.1.2:
      Successfully uninstalled pip-8.1.2
Successfully installed pip-19.3.1
+ pip install wheel
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting wheel
  Downloading https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.33.6
+ pip install -U setuptools
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting setuptools
  Downloading https://files.pythonhosted.org/packages/54/28/c45d8b54c1339f9644b87663945e54a8503cfef59cf0f65b3ff5dd17cf64/setuptools-42.0.2-py2.py3-none-any.whl (583kB)
Installing collected packages: setuptools
  Found existing installation: setuptools 0.9.8
    Uninstalling setuptools-0.9.8:
      Successfully uninstalled setuptools-0.9.8
Successfully installed setuptools-42.0.2
+ pip install 'vsc-install<0.11.4' 'vsc-base<2.9.0'
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting vsc-install<0.11.4
  Downloading https://files.pythonhosted.org/packages/18/59/3274a58af6af84a87f7655735b452c06c769586ee73954f5ee15d303aa29/vsc-install-0.11.3.tar.gz (61kB)
Collecting vsc-base<2.9.0
  Downloading https://files.pythonhosted.org/packages/48/aa/f05d350c358338d0e843835660e3993cc5eb28401f32c0c5b8bc9a9458d5/vsc-base-2.8.4.tar.gz (104kB)
Requirement already satisfied: setuptools in /usr/lib/python2.7/site-packages (from vsc-install<0.11.4) (42.0.2)
Building wheels for collected packages: vsc-install, vsc-base
  Building wheel for vsc-install (setup.py): started
  Building wheel for vsc-install (setup.py): finished with status 'done'
  Created wheel for vsc-install: filename=vsc_install-0.11.3-cp27-none-any.whl size=47520 sha256=4e9f0a622171b54db7dee8dfa25d8a0b01f919c1c209a5dfe15fb20d632abd3b
  Stored in directory: /root/.cache/pip/wheels/bf/51/2d/91126ff38375dd0c7e09d1a1a8981e936da190e829d63e5baa
  Building wheel for vsc-base (setup.py): started
  Building wheel for vsc-base (setup.py): finished with status 'done'
  Created wheel for vsc-base: filename=vsc_base-2.8.4-cp27-none-any.whl size=97828 sha256=d9e5eb6fc11b46cb1857517a65db971d251da9650b11534cd6434311b689465f
  Stored in directory: /root/.cache/pip/wheels/54/c1/f3/0e9e13a3589b8ab3f8fec0b1597a53638e44f38110c45953ef
Successfully built vsc-install vsc-base
Installing collected packages: vsc-install, vsc-base
Successfully installed vsc-base-2.8.4 vsc-install-0.11.3
+ pip install easybuild
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting easybuild
  Downloading https://files.pythonhosted.org/packages/05/b8/c8b3711ca4574f46cf6808072dadf703bdb620b144f7208094c289b1fd1d/easybuild-4.1.0.tar.gz
Collecting easybuild-framework==4.1.0
  Downloading https://files.pythonhosted.org/packages/75/70/0f4c795c8c16257f35ec677fb171c968f0bd10d4c144862d045d8b869ee0/easybuild-framework-4.1.0.tar.gz (1.8MB)
Collecting easybuild-easyblocks==4.1.0
  Downloading https://files.pythonhosted.org/packages/af/b5/627da5604c960ec688b64be6ac0ba09439865c9c2a45d40ed065f67132ab/easybuild-easyblocks-4.1.0.tar.gz (372kB)
Collecting easybuild-easyconfigs==4.1.0
  Downloading https://files.pythonhosted.org/packages/0e/03/1cf77cda33026d51e86df1092ced461ee51ab56cbfdd1d4633eddd9a36ec/easybuild-easyconfigs-4.1.0.tar.gz (4.0MB)
Building wheels for collected packages: easybuild, easybuild-framework, easybuild-easyblocks, easybuild-easyconfigs
  Building wheel for easybuild (setup.py): started
  Building wheel for easybuild (setup.py): finished with status 'done'
  Created wheel for easybuild: filename=easybuild-4.1.0-cp27-none-any.whl size=10196 sha256=7fff6fc09923db345881d2a42402154241e5418ad1d1c4a1580175d36aef47bb
  Stored in directory: /root/.cache/pip/wheels/12/8c/2f/240661ac59ade03438555ce5d6d7de1b7fd3c1fad96f6e3f32
  Building wheel for easybuild-framework (setup.py): started
  Building wheel for easybuild-framework (setup.py): finished with status 'done'
  Created wheel for easybuild-framework: filename=easybuild_framework-4.1.0-cp27-none-any.whl size=3191378 sha256=1aa5ba9e44c94e8c0881b17881afa3da7819742fdb64676f93ffbbb238dbf257
  Stored in directory: /root/.cache/pip/wheels/aa/ba/26/18547883df676a5d8b23aac0e430276eab02bb6120967d7beb
  Building wheel for easybuild-easyblocks (setup.py): started
  Building wheel for easybuild-easyblocks (setup.py): finished with status 'done'
  Created wheel for easybuild-easyblocks: filename=easybuild_easyblocks-4.1.0-cp27-none-any.whl size=629448 sha256=7ca64d637e6f2a0fa1f77835cd72ec4503862ca526ca73894ff27fc964deca95
  Stored in directory: /root/.cache/pip/wheels/da/0b/27/7ecd6e4055f4c5a2b3b3ea399b9b2476d0677b7acb38c5554b
  Building wheel for easybuild-easyconfigs (setup.py): started
  Building wheel for easybuild-easyconfigs (setup.py): finished with status 'done'
  Created wheel for easybuild-easyconfigs: filename=easybuild_easyconfigs-4.1.0-cp27-none-any.whl size=14416003 sha256=be2906e3c2b3758a7e80a46f180eeadb48a18270fcb6c9ea31480fc46533ff49
  Stored in directory: /root/.cache/pip/wheels/f1/ca/16/ed5dbf917499dab065c7662f3b6e48b9ad22dd8234f8781c7a
Successfully built easybuild easybuild-framework easybuild-easyblocks easybuild-easyconfigs
Installing collected packages: easybuild-framework, easybuild-easyblocks, easybuild-easyconfigs, easybuild
Successfully installed easybuild-4.1.0 easybuild-easyblocks-4.1.0 easybuild-easyconfigs-4.1.0 easybuild-framework-4.1.0
+ id easybuild
id: easybuild: no such user
+ useradd easybuild
+ '[' '!' -d /app ']'
+ mkdir -p /app
+ chown easybuild:easybuild -R /app
+ '[' '!' -d /scratch ']'
+ mkdir -p /scratch
+ chown easybuild:easybuild -R /scratch
+ cat
+ su - easybuild

# configure EasyBuild

# use /scratch as general prefix, used for sources, build directories, etc.
export EASYBUILD_PREFIX=/scratch

# also use /scratch for temporary directories
export EASYBUILD_TMPDIR=/scratch/tmp

# download sources to /scratch/sources, but also consider files located in /tmp/easybuild/sources;
# that way, source files that can not be downloaded can be seeded in
export EASYBUILD_SOURCEPATH=/scratch/sources:/tmp/easybuild/sources

# install software & modules into /app
export EASYBUILD_INSTALLPATH=/app

# use EasyBuild to install specified software
eb Anaconda3-5.3.0.eb --robot
== temporary log file in case of crash /scratch/tmp/eb-vsXdtZ/easybuild-qi5cCY.log
== resolving dependencies ...
== processing EasyBuild easyconfig /usr/easybuild/easyconfigs/a/Anaconda3/Anaconda3-5.3.0.eb
== building and installing Anaconda3/5.3.0...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== patching...
== preparing...
== configuring...
== building...
== testing...
== installing...
== taking care of extensions...
== restore after iterating...
== postprocessing...
== sanity checking...
== cleaning up...
== creating module...
== permissions...
== packaging...
== COMPLETED: Installation ended successfully (took 2 min 21 sec)
== Results of the build can be found in the log file(s) /app/software/Anaconda3/5.3.0/easybuild/easybuild-Anaconda3-5.3.0-20191217.193354.log
== Build succeeded for 1 out of 1
== Temporary log file(s) /scratch/tmp/eb-vsXdtZ/easybuild-qi5cCY.log* have been removed.
== Temporary directory /scratch/tmp/eb-vsXdtZ has been removed.

# update Lmod cache
mkdir -p /app/lmodcache
$LMOD_DIR/update_lmod_system_cache_files -d /app/lmodcache -t /app/lmodcache/timestamp /app/modules/all

# exit from 'easybuild' user
exit
# ~/.bash_logout

+ rm -rf /scratch/build /scratch/ebfiles_repo /scratch/sources /scratch/tmp
INFO:    Adding environment to container
INFO:    Adding runscript
INFO:    Creating SIF file...
INFO:    Build complete: /tmp/image-431409907
WARNING: Skipping container verifying
 1.67 GiB / 1.67 GiB  100.00% 97.35 MiB/s 17s01ss6s
INFO:    Build complete: Anaconda3-5.3.0.sif

```

Just confirming the container is built properly 

```
c455-002[knl](349)$ singularity exec Anaconda3-5.3.0.sif which conda
/app/software/Anaconda3/5.3.0/bin/conda
c455-002[knl](350)$ singularity exec Anaconda3-5.3.0.sif which python
/app/software/Anaconda3/5.3.0/bin/python
c455-002[knl](351)$ singularity exec Anaconda3-5.3.0.sif python -V
Python 3.7.0
c455-002[knl](352)$ python -V
Python 2.7.5

c455-002[knl](355)$ singularity shell --shell "/bin/bash" Anaconda3-5.3.0.sif
Singularity> type module
module is a function
module ()
{
    eval `$LMOD_CMD sh "$@"`
}
Singularity> module --version

Modules based on Lua: Version 8.2.7  2019-11-30 09:41 -06:00
    by Robert McLay mclay@tacc.utexas.edu

Singularity> module av

---------------------------------------------------------------------------------------------- /app/modules/all -----------------------------------------------------------------------------------------------
   Anaconda3/5.3.0 (L)

  Where:
   L:  Module is loaded

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".



```

I will test this with building few more containers to ensure the pip dependency is present in other containers. Then setting these commands in container recipe should be done from easybuild


Discussion:

Resolving patch:
diff --git a/easybuild/tools/containers/singularity.py b/easybuild/tools/containers/singularity.py
--- a/easybuild/tools/containers/singularity.py
+++ b/easybuild/tools/containers/singularity.py
@@ -311,6 +311,9 @@ def resolve_template_data(self):
         if 'install_eb' not in template_data:
             template_data['install_eb'] = '\n'.join([
                 "# install EasyBuild using pip",
+                # upgrade pip
+                "pip install -U pip",
+                "pip install wheel",
                 # EasyBuild 3.x requires setuptools as runtime dependency
                 "pip install -U setuptools",
                 # stick to previous version of vsc-install to avoid requiring mock (which causes installation problems)


Test patch: