Don't cache metadata in GCS connections
State: closed

Currently if you access a bucket, change the contents, and try to read a new file w/ e.g. `read_parquet`, you see a `FileNotFoundError`; restarting Python fixes this. IMO since the `gcsfs` connection is hidden from the user, caching metadata doesn't really make sense (how would you even go about clearing this manually?).

@martindurant seem ok to you?

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
The method you want is [invalidate_cache](https://gcsfs.readthedocs.io/en/latest/api.html#gcsfs.core.GCSFileSystem.invalidate_cache); this is automatically called if you do the writing with the same file-system instance.
This is a good opportunity for me to ask people's opinion on [fsspec](https://filesystem-spec.readthedocs.io/en/latest/) which will unify the APIs for all of these file-systems, and gcsfs will eventually depend on it. It would give a single place to define behaviour such as what is being discussed here, and obviate the requirement for this module altogether - functions in fsspec itself will deal with instantiating file-system instances and providing file-like objects.
@bnaul @martindurant : If you guys could open an issue to discuss all of this, that would be great.  IMO this PR seems a little premature given this discussion + the lack of an example (or test) that can be used to validate this fix.

What do you think?
@martindurant unifying all the different logics sounds great to me; I
wondered whether this also affected s3 but didn't get a chance to check.

I think a separate issue for fsspec would be great and I'm happy to weigh
in there, but IMO this in particular is more of a configuration issue than
a bug; if I had known about this behavior before I would have just passed
this value in my initial implementation of this logic and nobody would have
thought twice about it ;).

On Sat, Apr 27, 2019, 12:03 AM gfyoung <notifications@github.com> wrote:

> @bnaul <https://github.com/bnaul> @martindurant
> <https://github.com/martindurant> : If you guys could open an issue to
> discuss all of this, that would be great. IMO this PR seems a little
> premature given this discussion + the lack of an example (or test) that can
> be used to validate this fix.
>
> What do you think?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/26221#issuecomment-487261766>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAG4TZZXH2PCQIHSJVCIG5DPSP3EBANCNFSM4HI2NYVA>
> .
>

I'm surprised that there is no way to pass arguments to the GCSFileSystem constructor; it would seem reasonable to me to offer the choice to the user as to whether they want to use the listings cache, which offers efficiency of not repeatedly looking up the file listing, but does not notice new files added by third parties.

For s3fs, a "direct" method was added to get the info for just one file when being opened, if the directory had not been previously listed, and this could be an option for gcsfs too; another reason it would be nice to depend on fsspec, rather than having to duplicate such logic.
> I'm surprised that there is no way to pass arguments to the GCSFileSystem constructor; it would seem reasonable to me to offer the choice to the user as to whether they want to use the listings cache, which offers efficiency of not repeatedly looking up the file listing, but does not notice new files added by third parties.

we *already* have 50 args to the read_* routines, so not going to add anything further; you can simply open a connection and pass the handle if you want. eg.

``` 
fs = gcsfs.GCSFileSystem()
with fs.open(....) as fh:
     pd.read_*(fh)
```

so I am -1 on this PR, but would take a doc-example using alternative filesystems
@jreback that was exactly my thinking, that we don't want to clutter the
read_* arg space. hence wanting to use a more sensible default inside the
connection logic.

I don't feel strongly about this at all but it's puzzling to me that such a
minor change would be so objectionable to anyone; I don't see any reason to
prefer the current behavior to this and all the fsspec questions, while
definitely interesting per se, seem mostly separate from this

On Sun, Apr 28, 2019, 6:25 PM Jeff Reback <notifications@github.com> wrote:

> I'm surprised that there is no way to pass arguments to the GCSFileSystem
> constructor; it would seem reasonable to me to offer the choice to the user
> as to whether they want to use the listings cache, which offers efficiency
> of not repeatedly looking up the file listing, but does not notice new
> files added by third parties.
>
> we *already* have 50 args to the read_* routines, so not going to add
> anything further; you can simply open a connection and pass the handle if
> you want. eg.
>
> fs = gcsfs.GCSFileSystem()
> with fs.open(....) as fh:
>      pd.read_*(fh)
>
> so I am -1 on this PR, but would take a doc-example using alternative
> filesystems
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/26221#issuecomment-487393880>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAG4TZZV4SBVBRCLUTDG2NDPSXFXPANCNFSM4HI2NYVA>
> .
>

@bnaul if the default should be something else, then that should be in the upstream package itself (GCSFileSystem) as the default; pandas passing args here to change the default doesn't make much sense.
ok I can see that, thanks. feel free to close as I'm afk for the day

On Sun, Apr 28, 2019, 6:37 PM Jeff Reback <notifications@github.com> wrote:

> @bnaul <https://github.com/bnaul> if the default should be something
> else, then that should be in the upstream package itself (GCSFileSystem) as
> the default; pandas passing args here to change the default doesn't make
> much sense.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/26221#issuecomment-487394805>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAG4TZ4AJLBJJEG2T3I437DPSXHETANCNFSM4HI2NYVA>
> .
>

closing as the default should be per the upstream package.