Repository: spack/spack

Issue:
Error: attribute 'closed' of '_io._IOBase' objects is not writable
### Steps to reproduce

The error occurs in 

```bash
cat > Vagrantfile << \EOF
Vagrant.configure("2") do |config|
  config.vm.box = "qnib/rocky8-spack"
  config.vm.box_version = "2022.02.08-v2"
end
EOF
vagrant up
```

This spins up a virtual machine with spack installed.

```bash
$ vagrant ssh
[vagrant@localhost ~]$ sudo su - spack
[spack@localhost ~]$
```

I followed the following steps to create a binary cache.
**Note**: The S3 bucket is already created using `aws s3 mb s3://name`.

## Binary cache creation

Let's create a binary cache in AWS S3 and upload some packages to show the benefit.

```bash
export GPG_KEY=qnib-binary-cache-key
export BUILD_CACHE=spack-vagrant-cache
export GPG_EMAIL=spack@test.com
```
### Create S3 Bucket

Log into your AWS console and create a bucket with the name specified above.

First, we need to configure the `awscli` with our AWS credentials. 

```bash
aws configure
```

Make sure to use a IAM profile that is restricted to only read and write to the bucket in question.

```bash
$ aws configure
AWS Access Key ID [None]: <YOUR_ACCESS_KEY>
AWS Secret Access Key [None]: <YOUR_SECRET_KEY>
Default region name [None]: eu-west-1
Default output format [None]:
```

### GPG Keys

```bash
spack gpg init
spack gpg create ${GPG_KEY} ${GPG_EMAIL}
```
This will create a key within ``.

```bash
$ spack gpg create ${GPG_KEY} ${GPG_EMAIL}
gpg: keybox '/opt/spack/opt/spack/gpg/pubring.kbx' created
gpg: /opt/spack/opt/spack/gpg/trustdb.gpg: trustdb created
gpg: key TEST123 marked as ultimately trusted
gpg: directory '/opt/spack/opt/spack/gpg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/opt/spack/opt/spack/gpg/openpgp-revocs.d/TEST123.rev'```

### Create Mirror

```bash
spack mirror add ${BUILD_CACHE} "s3://${BUILD_CACHE}"
spack mirror list
```

A mirror should be added to your local `~/.spack/mirrors.yaml`.


```bash
$ spack mirror list
spack-vagrant-cache    s3://spack-vagrant-cache (fetch)
spack-vagrant-cache    s3://spack-vagrant-cache (push)
spack-public           https://mirror.spack.io
$
```

### Install Package

```bash
spack install bzip2
```

### Fill Cache

```bash
spack buildcache create -f --rebuild-index -k ${GPG_KEY} -m ${BUILD_CACHE} bzip2
```
### Use Cache

The mirror we added above is per user, by changing the user we can utilize the mirror.

```bash
export BUILD_CACHE=spack-vagrant-bins
spack mirror add ${BUILD_CACHE} "s3://${BUILD_CACHE}"
spack install --no-check-signature --cache-only bzip2
```

### Error message

OK, that the setup...

I am encountering this error. Sometimes when I am installing software...

```bash
[spack@localhost ~]$ spack install bzip2
[+] /nfs/share/spack/pkg/linux-rocky8-x86_64/gcc-8.5.0/libiconv-1.16-35blyeuqkakrjcje6b7rpi52bylxmjoc
[+] /nfs/share/spack/pkg/linux-rocky8-x86_64/gcc-8.5.0/diffutils-3.8-ku4gppnjmbo2e6zdcb6ht2nzjdid2ac7
==> Installing bzip2-1.0.8-pbg6ou5bp6d3ob66fvd4ymetnbkegtgj
==> Error: Failed to install bzip2 due to AttributeError: attribute 'closed' of '_io._IOBase' objects is not writable
==> Error: attribute 'closed' of '_io._IOBase' objects is not writable
[spack@localhost ~]$
```

Sometimes when I want to upload to the S3 bucket.

```bash
$ spack buildcache create -f --rebuild-index -k ${GPG_KEY} -m ${BUILD_CACHE} bzip2
==> Pushing binary packages to s3://spack-vagrant-cache/build_cache
gpg: using "qnib-binary-cache-key" as default secret key for signing
==> Error: attribute 'closed' of '_io._IOBase' objects is not writable
```

### Information on your system

```bash
$ spack debug report
* **Spack:** 0.17.1-1106-c1b51d6e99
* **Python:** 3.6.8
* **Platform:** linux-rocky8-sandybridge
* **Concretizer:** clingo
```

**Note**: Since my setup should be only used to to dryrun testing I specified `x86_64` in the `packages.yaml`.

```bash
$ cat /opt/spack/etc/spack/packages.yaml
packages:
  all:
    target: [x86_64]
  slurm:
    buildable: false
    externals:
    - spec: slurm@21-08-5-1
      prefix: /usr
  openmpi:
    variants: +pmi +legacylaunchers schedulers=slurm
    version:
       - 4.1.0
$ cat /opt/spack/etc/spack/config.yaml |grep -v "#"
config:
  install_tree:
    root: /nfs/share/spack/pkg
  source_cache: /nfs/share/spack/cache
  misc_cache: ${HOME}/cache
```

```bash
$ cat /opt/spack/etc/spack/compilers.yaml
compilers:
- compiler:
    spec: gcc@8.5.0
    paths:
      cc: /bin/gcc
      cxx: /bin/g++
      f77: /bin/gfortran
      fc: /bin/gfortran
    flags: {}
    operating_system: rocky8
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
```

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
@alalazo @opadron As discussed in Slack
How much memory does Spack need.. Might it be an issue with the size of RAM? I usually grant 2GB to the headnode in the Vagrant stack).
The above gets the default (which I think is 1GB) - and it crashes even earlier...

**UPDATE**: I guess not. Even with 4GB I get the error and the memory usage does not even reach 1GB.
I have seen this error consistently over the last week. It has occurred on >=8GB VMs, containers, and bare metal, with Rocky, Fedora35 and x86_64 and AArch64. I have observed it with develop, v0.17.0, v0.17.1

I make heavy use of  S3 buckets for caching - which has worked well in the past but seems to be the area of concern now?
You have seen this as well and you do not think it is a memory issue?
Any workaround you figured so far?
I'll try Alma Linux then..
Same with Alma Linux 8..
```
$ spack buildcache create -f --rebuild-index -k ${GPG_KEY} -m ${BUILD_CACHE} bzip2
==> Pushing binary packages to s3://spack-vagrant-cache/build_cache
gpg: using "qnib-binary-cache-key" as default secret key for signing
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
==> Error: attribute 'closed' of '_io._IOBase' objects is not writable
[spack@localhost ~]$ cat /etc/os-release
NAME="AlmaLinux"
VERSION="8.5 (Arctic Sphynx)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="AlmaLinux 8.5 (Arctic Sphynx)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:almalinux:almalinux:8::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-8"
ALMALINUX_MANTISBT_PROJECT_VERSION="8.5"

[spack@localhost ~]$
```

I get the same error with 0.16.3 on Rocky Linux 8. :( 
I am afraid, that error is above my expertise - I might have messed something up.
I don't have a work around yet.

This issue suddenly started occurring for me last week.

I feel like for me this is not a lack-of-memory issue as I have observed this issue on some large bare metal servers. IOBase is apparently a base class for Python IO. My feeling at this point, with the evidence I have, is it feels like this is somehow system related - as opposed to spack related (I tested three version of spack and they all experience the issue).

My current focus on on my system's interaction with a s3 bucket (which I use heavily).
I tried Rocky8 and Alma8 - now switching to Amazon Linux 2 (all on virtualbox using Vagrant). Let's see... :) 
Geezes, ALinux2 (CentOS 7 based) has the same issue. Might that even be a problem with the VM setup? Or some security foo stepping in?
it looks like on my system, the s3 bucket may be involved:

```
spack --debug -vvvvvvvvvv buildcache update-index -d s3:my-spack-binary-cachexxxx/v0.17.0/
...
==> [2022-02-09-18:19:34.333597] fetching s3://my-spack-binary-cachexxxx/v0.17.0/build_cache/linux-centos8-thunderx2-gcc-8.3.1-gcc-10.2.0-afr62nir3m27qv7sryzllp4slmdt26dj.spec.json                                                     
Traceback (most recent call last):
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/util/web.py", line 552, in _urlopen                                                                                                      
    return opener(req, *args, **kwargs)
TypeError: open() got an unexpected keyword argument 'context'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./job_spack/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/main.py", line 881, in main                                                                                                              
    return _main(argv)
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/main.py", line 864, in _main                                                                                                             
    return _invoke_command(command, parser, args, unknown)
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/main.py", line 534, in _invoke_command                                                                                                   
    return_val = command(parser, args)
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/cmd/buildcache.py", line 993, in buildcache                                                                                              
    args.func(args)
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/cmd/buildcache.py", line 988, in buildcache_update_index                                                                                 
    update_index(outdir, update_keys=args.keys)
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/cmd/buildcache.py", line 972, in update_index                                                                                            
    url_util.join(outdir, bindist.build_cache_relative_path()))
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/binary_distribution.py", line 795, in generate_package_index                                                                             
    _, _, spec_file = web_util.read_from_url(spec_url)
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/util/web.py", line 132, in read_from_url                                                                                                 
    response = _urlopen(req, timeout=_timeout, context=context)
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/util/web.py", line 557, in _urlopen                                                                                                      
    return opener(req, *args, **kwargs)
  File "/usr/lib64/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/lib64/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/lib64/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/s3_handler.py", line 66, in s3_open                                                                                                      
    url, headers, stream = _s3_open(orig_url)
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/s3_handler.py", line 55, in _s3_open                                                                                                     
    stream = WrapStream(obj['Body'])
  File "/home/gitlab-runner/builds/2QDZ1uPi/0/engineering/foundry/foundry/job_spack/lib/spack/spack/s3_handler.py", line 28, in __init__                                                                                                     
    raw.closed = False
AttributeError: attribute 'closed' of '_io._IOBase' objects is not writable
```
In my case, I'm setting `url_fetch_method:curl` (for a reason I can't remember). This issue may be related to: https://github.com/spack/spack/issues/28753 ?
As a blind hack to see if I could get my system running again, I commented out[ line 29](https://github.com/spack/spack/blob/7f1759b52eeffdd67d55cb4af95011415e3d21ed/lib/spack/spack/s3_handler.py#L29) in `lib/spack/spack/s3_handler.py`. The system appears to be running now, and reading/writing from my s3 bucket as expected?
Might happen to be boto3's fault...
```
pip3 install boto3==1.20.35
pip3 install botocore==1.23.42
```
Fixes the issue for me... 👯‍♂️ 

(Randomly choosing versions from 3 to 4 weeks ago)
ok, two solutions... 😅 
IOBase was implemented in botocore on 12/31: https://github.com/boto/botocore/commit/a624815eabac50442ed7404f3c4f2664cd0aa784
I prefer your workaround :)
On my Amazon Linux 2 box, the culprit seems to be `botocore`. When I move from `1.23.46` to `1.23.47` it breaks for me.
```
Installing collected packages: botocore
  Attempting uninstall: botocore
    Found existing installation: botocore 1.23.46
    Uninstalling botocore-1.23.46:
      Successfully uninstalled botocore-1.23.46
Successfully installed botocore-1.23.47
Thanks for the detective work.  I'm guessing that code I wrote was working around a botocore issue that may have since been fixed?
@opadron I suppose you need to fix the dependency to botocore. Should not be later than 1.23.46.
Downgrading botocore version per @ChristianKniep suggestion corrected the issue for me too. Thank you.
Ran into this as well. Will rebuild gitlab-runner to lock this down
Can we please update the code to work with new botocore instead of freezing it?
Do these versions still need to be locked? It has been a while. 

```
pip3 install botocore==1.23.42 boto3==1.20.35
```

> Do these versions still need to be locked? It has been a while.
> 
> ```
> pip3 install botocore==1.23.42 boto3==1.20.35
> ```

I still hit this issue when trying to run `spack buildcache update-index` on an S3 endpoint, if the versions aren't pinned:

```
root@b3429183af47:/# pip list installed
Package            Version
------------------ ---------
boto3              1.24.26
botocore           1.27.26
...

root@b3429183af47:/# spack buildcache update-index -d s3://cache.e4s.io/zzz
==> Warning: Encountered problem pushing package index to s3://cache.e4s.io/zzz/build_cache: attribute 'closed' of '_io._IOBase' objects is not writable
```

Resolving patch:
diff --git a/lib/spack/spack/s3_handler.py b/lib/spack/spack/s3_handler.py
--- a/lib/spack/spack/s3_handler.py
+++ b/lib/spack/spack/s3_handler.py
@@ -3,7 +3,7 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
-from io import BufferedReader
+from io import BufferedReader, IOBase
 
 import six
 import six.moves.urllib.error as urllib_error
@@ -23,11 +23,15 @@
 # https://github.com/python/cpython/pull/3249
 class WrapStream(BufferedReader):
     def __init__(self, raw):
-        raw.readable = lambda: True
-        raw.writable = lambda: False
-        raw.seekable = lambda: False
-        raw.closed = False
-        raw.flush = lambda: None
+        # In botocore >=1.23.47, StreamingBody inherits from IOBase, so we
+        # only add missing attributes in older versions.
+        # https://github.com/boto/botocore/commit/a624815eabac50442ed7404f3c4f2664cd0aa784
+        if not isinstance(raw, IOBase):
+            raw.readable = lambda: True
+            raw.writable = lambda: False
+            raw.seekable = lambda: False
+            raw.closed = False
+            raw.flush = lambda: None
         super(WrapStream, self).__init__(raw)
 
     def detach(self):


Test patch: