Repository: pandas-dev/pandas
Pull request: IO: Fix S3 Error Handling
closes #27679 
closes #32486 

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
to_parquet swallows NoCredentialsError
#### Code Sample, a copy-pastable example if possible

```python
try: 
   pd.DataFrame({'foo': [None, ['foo', 'bar']]}).to_parquet('s3://foo/bar') 
except Exception as e: 
   print('Here')                                                                                                                                                                                                            
```
#### Problem description

Without credentials configured, the above code does not write any output, but also doesn't end up in the exception handling code. Instead, on stdout or stderr, an exception is printed:

```
Exception ignored in: <function AbstractBufferedFile.__del__ at 0x7fe0ae8db440>
Traceback (most recent call last):
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/fsspec/spec.py", line 1137, in __del__
    self.close()
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/fsspec/spec.py", line 1114, in close
    self.flush(force=True)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/fsspec/spec.py", line 986, in flush
    self._initiate_upload()
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/s3fs/core.py", line 951, in _initiate_upload
    Bucket=self.bucket, Key=self.key, ACL=self.acl)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/s3fs/core.py", line 939, in _call_s3
    **kwargs)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/s3fs/core.py", line 182, in _call_s3
    return method(**additional_kwargs)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/client.py", line 648, in _make_api_call
    operation_model, request_dict, request_context)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/signers.py", line 157, in sign
    auth.add_auth(request)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/auth.py", line 425, in add_auth
    super(S3SigV4Auth, self).add_auth(request)
  File "/home/languitar/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
```

#### Expected Output

`Here`

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.4.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.19.61-1-lts
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.0
numpy            : 1.17.0
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 19.2.1
setuptools       : 41.0.1
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : 1.1.8
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.1
IPython          : 7.7.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.14.1
pytables         : None
s3fs             : 0.3.1
scipy            : None
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : 1.1.8

</details>


to_csv fails silently with s3fs
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))
df.to_csv("s3://super_dumb_s3_bucket/not_real.csv")

```
#### Problem description

If you provide an invalid s3 path (e.g. a bucket that you don't have access to), this command fails silently.

#### Expected Output

Raise some exception at least.

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.2.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.15.0-88-generic
machine          : x86_64
processor        : 
byteorder        : little
LC_ALL           : None
LANG             : C.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 0.25.1
numpy            : 1.17.0
pytz             : 2019.2
dateutil         : 2.8.0
pip              : 19.0.2
setuptools       : 40.8.0
Cython           : None
pytest           : 3.10.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.4.1
html5lib         : None
pymysql          : None
psycopg2         : 2.8.3 (dt dec pq3 ext lo64)
jinja2           : 2.10.1
IPython          : 7.9.0
pandas_datareader: None
bs4              : 4.8.0
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.4.1
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.1
sqlalchemy       : None
tables           : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>

Hints from issue discussion:
Based on `Exception ignored in: <function AbstractBufferedFile.__del__ at 0x7fe0ae8db440>`

this is most likely an issue in s3fs / fsspec, right? Or do you think it's in how pandas is calling things?
I have no idea who is responsible for this.
Thanks. Let us know if you're interested in looking further.

On Thu, Aug 1, 2019 at 2:06 AM Johannes Wienke <notifications@github.com>
wrote:

> I have no idea who is responsible for this.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/issues/27679?email_source=notifications&email_token=AAKAOIRAAJJ6EK3VTOGO35LQCKDPDA5CNFSM4IIIB6NKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3JRN3Y#issuecomment-517150447>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIQ4GRZPIV7BVBUMA7DQCKDPDANCNFSM4IIIB6NA>
> .
>

Using `S3FileSystem` directly with `anon=False` and no credentials correctly raises the exception. So it must be something around in pandas that is swallowing this:
```
In [4]: s3fs.S3FileSystem(anon=False).open('/test/bar')                                                                                                                                                             
---------------------------------------------------------------------------
NoCredentialsError                        Traceback (most recent call last)
<ipython-input-4-04d546643798> in <module>
----> 1 s3fs.S3FileSystem(anon=False).open('/test/bar')

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/fsspec/spec.py in open(self, path, mode, block_size, **kwargs)
    658             ac = kwargs.pop('autocommit', not self._intrans)
    659             f = self._open(path, mode=mode, block_size=block_size,
--> 660                            autocommit=ac, **kwargs)
    661             if not ac:
    662                 self.transaction.files.append(f)

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/s3fs/core.py in _open(self, path, mode, block_size, acl, version_id, fill_cache, cache_type, autocommit, **kwargs)
    301                       version_id=version_id, fill_cache=fill_cache,
    302                       s3_additional_kwargs=kw, cache_type=cache_type,
--> 303                       autocommit=autocommit)
    304 
    305     def _lsdir(self, path, refresh=False, max_items=None):

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/s3fs/core.py in __init__(self, s3, path, mode, block_size, acl, version_id, fill_cache, s3_additional_kwargs, autocommit, cache_type)
    913         self.s3_additional_kwargs = s3_additional_kwargs or {}
    914         super().__init__(s3, path, mode, block_size, autocommit=autocommit,
--> 915                          cache_type=cache_type)
    916         if self.writable():
    917             if block_size < 5 * 2 ** 20:

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/fsspec/spec.py in __init__(self, fs, path, mode, block_size, autocommit, cache_type, **kwargs)
    853         if mode == 'rb':
    854             if not hasattr(self, 'details'):
--> 855                 self.details = fs.info(path)
    856             self.size = self.details['size']
    857             self.cache = caches[cache_type](self.blocksize, self._fetch_range,

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/s3fs/core.py in info(self, path, version_id)
    472             except ParamValidationError as e:
    473                 raise ValueError('Failed to head path %r: %s' % (path, e))
--> 474         return super().info(path)
    475 
    476     def ls(self, path, detail=False, refresh=False, **kwargs):

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/fsspec/spec.py in info(self, path, **kwargs)
    466         """
    467         path = self._strip_protocol(path)
--> 468         out = self.ls(self._parent(path), detail=True, **kwargs)
    469         out = [o for o in out if o['name'].rstrip('/') == path]
    470         if out:

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/s3fs/core.py in ls(self, path, detail, refresh, **kwargs)
    490         """
    491         path = self._strip_protocol(path).rstrip('/')
--> 492         files = self._ls(path, refresh=refresh)
    493         if not files:
    494             files = self._ls(self._parent(path), refresh=refresh)

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/s3fs/core.py in _ls(self, path, refresh)
    428             return self._lsbuckets(refresh)
    429         else:
--> 430             return self._lsdir(path, refresh)
    431 
    432     def exists(self, path):

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/s3fs/core.py in _lsdir(self, path, refresh, max_items)
    319                 files = []
    320                 dircache = []
--> 321                 for i in it:
    322                     dircache.extend(i.get('CommonPrefixes', []))
    323                     for c in i.get('Contents', []):

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/paginate.py in __iter__(self)
    253         self._inject_starting_params(current_kwargs)
    254         while True:
--> 255             response = self._make_request(current_kwargs)
    256             parsed = self._extract_parsed_response(response)
    257             if first_request:

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/paginate.py in _make_request(self, current_kwargs)
    330 
    331     def _make_request(self, current_kwargs):
--> 332         return self._method(**current_kwargs)
    333 
    334     def _extract_parsed_response(self, response):

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/client.py in _api_call(self, *args, **kwargs)
    355                     "%s() only accepts keyword arguments." % py_operation_name)
    356             # The "self" in this scope is referring to the BaseClient.
--> 357             return self._make_api_call(operation_name, kwargs)
    358 
    359         _api_call.__name__ = str(py_operation_name)

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/client.py in _make_api_call(self, operation_name, api_params)
    646         else:
    647             http, parsed_response = self._make_request(
--> 648                 operation_model, request_dict, request_context)
    649 
    650         self.meta.events.emit(

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/client.py in _make_request(self, operation_model, request_dict, request_context)
    665     def _make_request(self, operation_model, request_dict, request_context):
    666         try:
--> 667             return self._endpoint.make_request(operation_model, request_dict)
    668         except Exception as e:
    669             self.meta.events.emit(

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/endpoint.py in make_request(self, operation_model, request_dict)
    100         logger.debug("Making request for %s with params: %s",
    101                      operation_model, request_dict)
--> 102         return self._send_request(request_dict, operation_model)
    103 
    104     def create_request(self, params, operation_model=None):

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/endpoint.py in _send_request(self, request_dict, operation_model)
    130     def _send_request(self, request_dict, operation_model):
    131         attempts = 1
--> 132         request = self.create_request(request_dict, operation_model)
    133         context = request_dict['context']
    134         success_response, exception = self._get_response(

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/endpoint.py in create_request(self, params, operation_model)
    114                 op_name=operation_model.name)
    115             self._event_emitter.emit(event_name, request=request,
--> 116                                      operation_name=operation_model.name)
    117         prepared_request = self.prepare_request(request)
    118         return prepared_request

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/hooks.py in emit(self, event_name, **kwargs)
    354     def emit(self, event_name, **kwargs):
    355         aliased_event_name = self._alias_event_name(event_name)
--> 356         return self._emitter.emit(aliased_event_name, **kwargs)
    357 
    358     def emit_until_response(self, event_name, **kwargs):

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/hooks.py in emit(self, event_name, **kwargs)
    226                  handlers.
    227         """
--> 228         return self._emit(event_name, kwargs)
    229 
    230     def emit_until_response(self, event_name, **kwargs):

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/hooks.py in _emit(self, event_name, kwargs, stop_on_response)
    209         for handler in handlers_to_call:
    210             logger.debug('Event %s: calling handler %s', event_name, handler)
--> 211             response = handler(**kwargs)
    212             responses.append((handler, response))
    213             if stop_on_response and response is not None:

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/signers.py in handler(self, operation_name, request, **kwargs)
     88         # this method is invoked to sign the request.
     89         # Don't call this method directly.
---> 90         return self.sign(operation_name, request)
     91 
     92     def sign(self, operation_name, request, region_name=None,

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/signers.py in sign(self, operation_name, request, region_name, signing_type, expires_in, signing_name)
    155                     raise e
    156 
--> 157             auth.add_auth(request)
    158 
    159     def _choose_signer(self, operation_name, signing_type, context):

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/auth.py in add_auth(self, request)
    423         self._region_name = signing_context.get(
    424             'region', self._default_region_name)
--> 425         super(S3SigV4Auth, self).add_auth(request)
    426 
    427     def _modify_request_before_signing(self, request):

~/.pyenv/versions/analytics-3.7/lib/python3.7/site-packages/botocore/auth.py in add_auth(self, request)
    355     def add_auth(self, request):
    356         if self.credentials is None:
--> 357             raise NoCredentialsError
    358         datetime_now = datetime.datetime.utcnow()
    359         request.context['timestamp'] = datetime_now.strftime(SIGV4_TIMESTAMP)

NoCredentialsError: Unable to locate credentials
```
Pandas does do a try / except in io.s3.get_filepath_or_buffer. So if you
don't have creds, we fall back to doing an anon=True request.

But that doesn't work for the write in my case. So nothing is written and no exception is raised.
Likely related to https://github.com/pandas-dev/pandas/issues/32486 https://github.com/pandas-dev/pandas/issues/32470
Hmm...  thanks for the report.

I think the issue is the failure to call close in https://github.com/pandas-dev/pandas/blob/970499d9660a4f4a1df3e1f8ce3015336ffb4d41/pandas/io/formats/csvs.py#L174-L204. S3FS only makes the request to validate the credentials when the file is actually closed, which we don't seem to do here.

Are you interested in debugging further?
👍  just ran into this myself. very dangerous silent failure..
See also https://github.com/pandas-dev/pandas/issues/32470. Can anyone see if reverting the relevant changes in https://github.com/pandas-dev/pandas/commit/d6fe1940a5d7ea10624e1cc871a6eef13f32b382#diff-a37b395bed03f0404dec864a4529c97d (at least for write mode) is doable?

Commit message: Add whatsnew

Patch to review:
diff --git a/doc/source/whatsnew/v1.1.0.rst b/doc/source/whatsnew/v1.1.0.rst
index 03a547fadd7ca..9347bb6b8a22e 100644
--- a/doc/source/whatsnew/v1.1.0.rst
+++ b/doc/source/whatsnew/v1.1.0.rst
@@ -582,6 +582,8 @@ I/O
 - Bug in :func:`pandas.io.json.json_normalize` where location specified by `record_path` doesn't point to an array. (:issue:`26284`)
 - :func:`pandas.read_hdf` has a more explicit error message when loading an
   unsupported HDF file (:issue:`9539`)
+- Bug in :meth:`to_parquet` was not raising ``PermissionError`` when writing to a private s3 bucket with invalid creds. (:issue:`27679`)
+- Bug in :meth:`to_csv` was silently failing when writing to an invalid s3 bucket. (:issue:`32486`)
 
 Plotting
 ^^^^^^^^
diff --git a/pandas/io/formats/csvs.py b/pandas/io/formats/csvs.py
index 091f7662630ff..dcd764bec7426 100644
--- a/pandas/io/formats/csvs.py
+++ b/pandas/io/formats/csvs.py
@@ -62,7 +62,7 @@ def __init__(
         # Extract compression mode as given, if dict
         compression, self.compression_args = get_compression_method(compression)
 
-        self.path_or_buf, _, _, _ = get_filepath_or_buffer(
+        self.path_or_buf, _, _, self.should_close = get_filepath_or_buffer(
             path_or_buf, encoding=encoding, compression=compression, mode=mode
         )
         self.sep = sep
@@ -223,6 +223,8 @@ def save(self) -> None:
                 f.close()
                 for _fh in handles:
                     _fh.close()
+            elif self.should_close:
+                f.close()
 
     def _save_header(self):
         writer = self.writer
diff --git a/pandas/io/parquet.py b/pandas/io/parquet.py
index 33747d2a6dd83..068210eddcc1b 100644
--- a/pandas/io/parquet.py
+++ b/pandas/io/parquet.py
@@ -92,7 +92,7 @@ def write(
         **kwargs,
     ):
         self.validate_dataframe(df)
-        path, _, _, _ = get_filepath_or_buffer(path, mode="wb")
+        path, _, _, should_close = get_filepath_or_buffer(path, mode="wb")
 
         from_pandas_kwargs: Dict[str, Any] = {"schema": kwargs.pop("schema", None)}
         if index is not None:
@@ -109,6 +109,8 @@ def write(
             )
         else:
             self.api.parquet.write_table(table, path, compression=compression, **kwargs)
+        if should_close:
+            path.close()
 
     def read(self, path, columns=None, **kwargs):
         path, _, _, should_close = get_filepath_or_buffer(path)
diff --git a/pandas/tests/io/parser/test_network.py b/pandas/tests/io/parser/test_network.py
index b7164477c31f2..cb958c1735be0 100644
--- a/pandas/tests/io/parser/test_network.py
+++ b/pandas/tests/io/parser/test_network.py
@@ -159,7 +159,7 @@ def test_parse_public_s3_bucket_nrows_python(self, tips_df):
             assert not df.empty
             tm.assert_frame_equal(tips_df.iloc[:10], df)
 
-    def test_s3_fails(self):
+    def test_read_s3_fails(self):
         with pytest.raises(IOError):
             read_csv("s3://nyqpug/asdf.csv")
 
@@ -168,6 +168,20 @@ def test_s3_fails(self):
         with pytest.raises(IOError):
             read_csv("s3://cant_get_it/file.csv")
 
+    def test_write_s3_csv_fails(self, tips_df):
+        # GH 32486
+        with pytest.raises(
+            FileNotFoundError, match="The specified bucket does not exist"
+        ):
+            tips_df.to_csv("s3://an_s3_bucket_data_doesnt_exit/not_real.csv")
+
+    def test_write_s3_parquet_fails(self, tips_df):
+        # GH 27679
+        with pytest.raises(
+            FileNotFoundError, match="The specified bucket does not exist"
+        ):
+            tips_df.to_parquet("s3://an_s3_bucket_data_doesnt_exit/not_real.parquet")
+
     def test_read_csv_handles_boto_s3_object(self, s3_resource, tips_file):
         # see gh-16135
 
diff --git a/pandas/tests/io/test_gcs.py b/pandas/tests/io/test_gcs.py
index 557a9d5c13987..cf745fcc492a1 100644
--- a/pandas/tests/io/test_gcs.py
+++ b/pandas/tests/io/test_gcs.py
@@ -56,7 +56,15 @@ def open(*args):
 
     monkeypatch.setattr("gcsfs.GCSFileSystem", MockGCSFileSystem)
     df1.to_csv("gs://test/test.csv", index=True)
-    df2 = read_csv(StringIO(s.getvalue()), parse_dates=["dt"], index_col=0)
+
+    def mock_get_filepath_or_buffer(*args, **kwargs):
+        return StringIO(df1.to_csv()), None, None, False
+
+    monkeypatch.setattr(
+        "pandas.io.gcs.get_filepath_or_buffer", mock_get_filepath_or_buffer
+    )
+
+    df2 = read_csv("gs://test/test.csv", parse_dates=["dt"], index_col=0)
 
     tm.assert_frame_equal(df1, df2)
 
@@ -86,28 +94,6 @@ def open(self, path, mode="r", *args):
     )
 
 
-@td.skip_if_no("gcsfs")
-def test_gcs_get_filepath_or_buffer(monkeypatch):
-    df1 = DataFrame(
-        {
-            "int": [1, 3],
-            "float": [2.0, np.nan],
-            "str": ["t", "s"],
-            "dt": date_range("2018-06-18", periods=2),
-        }
-    )
-
-    def mock_get_filepath_or_buffer(*args, **kwargs):
-        return (StringIO(df1.to_csv(index=False)), None, None, False)
-
-    monkeypatch.setattr(
-        "pandas.io.gcs.get_filepath_or_buffer", mock_get_filepath_or_buffer
-    )
-    df2 = read_csv("gs://test/test.csv", parse_dates=["dt"])
-
-    tm.assert_frame_equal(df1, df2)
-
-
 @td.skip_if_installed("gcsfs")
 def test_gcs_not_present_exception():
     with pytest.raises(ImportError) as e:


Code review comments:
diff_hunk: @@ -582,6 +582,8 @@ I/O
 - Bug in :func:`pandas.io.json.json_normalize` where location specified by `record_path` doesn't point to an array. (:issue:`26284`)
 - :func:`pandas.read_hdf` has a more explicit error message when loading an
   unsupported HDF file (:issue:`9539`)
+- Bug in :meth:`to_parquet` was not raising ``PermissionError`` when writing to a private s3 bucket with invalid creds. (:issue:`27679`)
+- Bug in :meth:`to_csv` was silently failing when writing to an invalid s3 bucket. (:issue:`32486`)
path: doc/source/whatsnew/v1.1.0.rst
text: @user1:
```suggestion
- Bug in :meth:`~DataFrame.to_parquet` was not raising ``PermissionError`` when writing to a private s3 bucket with invalid creds. (:issue:`27679`)
- Bug in :meth:`~DataFrame.to_csv` was silently failing when writing to an invalid s3 bucket. (:issue:`32486`)
```

diff_hunk: @@ -168,6 +168,20 @@ def test_s3_fails(self):
         with pytest.raises(IOError):
             read_csv("s3://cant_get_it/file.csv")
 
+    def test_write_s3_csv_fails(self, tips_df):
+        # GH 32486
+        with pytest.raises(
+            FileNotFoundError, match="The specified bucket does not exist"
+        ):
+            tips_df.to_csv("s3://an_s3_bucket_data_doesnt_exit/not_real.csv")
+
+    def test_write_s3_parquet_fails(self, tips_df):
+        # GH 27679
path: pandas/tests/io/parser/test_network.py
text: @user1:
Does this need an importorskip, or do we raise prior to importing the engine?

@author:
Added an importskip. Would raise ImportError if PyArrow or fastparquet isn’t installed.

@user2:
can you add the decorator version instead 

@author:
have done - and changed other importerskip in this file