opened
poudro
username_0
Been having a strange issue with `fs.open` on various files. 

The code is basically
```
    if not fs.exists(path_in_fs):
        return None

    try:
        with fs.open(path_in_fs, 'rb') as f:
            <- read file
    except Exception as err:
        raise err
```

Sometimes `fs.exists` will say that a file exists in s3 but the `fs.open` step will return a `ResourceNotFound` error.

I tracked the error in `fs.open` to this section of the code https://github.com/PyFilesystem/s3fs/blob/master/fs_s3fs/_s3fs.py#L433-L441, but I'm a bit at a loss as to why an exception should be triggered here but not for the `fs.exists`.

Finally, if I delete the file via other means (awscli or in s3 browser), and recreate the file (by copying a local copy to s3 via awscli), it will then work without triggering an error.

Any help on what I'm doing wrong would be greatly appreciated.

Because of this last element it's a bit hard to reproduce, but if you need more details please let me know.
`fs.open` returns `ResourceNotFound` (404) even though `fs.exists` finds the path
issue
created
poudro
username_0
I was able to reproduce, seems like if some other user wrote the file this occurs.

If I comment out line 439 it works as expected.
comment
created
shadiakiki1986
username_1
I ran in to this issue and investigated a bit. 

TL;DR: This behavior is intended by `s3fs` and was introduced as a fix to issue #17 in PR #21

It turns out that it's related to how S3 handles the directories that represent the path to the file. S3 is not really a filesystem as it doesn't create the directory structure to the file, but only stores the full path as a key. So if you upload to s3 something like `s3://bucket/folder/file.txt` **using aw scli**, then `/folder` doesn't exist in itself, whereas on a regular filesystem (say in my ubuntu terminal) it would. For `s3fs` in particular, and since it is part of `PyFilesystem`, it has to comply with how other filesystems work, so it does some extra stuff that would not be done if you were using the aws cli. For example, if you upload a file to s3 with s3fs to `s3://bucket/folder/file.txt`, it would create both `/folder/` and `/folder/file.txt`. At the same time, if you read that same path **after** having created it with `s3fs`, then the above issue won't happen. That's because the `obj.load` (that you commented out in PR #71 ) is verifying that the `/folder/` path exists, as would make sense in a normal filesystem. However, if you read that path **after** having uploaded it with aws cli for example, then the above issue will happen because the aws cli doesn't create `/folder/`. The `s3fs` way to handle this would be to use the `dir_path` argument in the `S3FS` constructor. Here is a full example to illustrate in code what I blaberred about in the prose above:

```
from fs_s3fs import S3FS
from zipfile import ZipFile

s3fs = S3FS('dolphicom')
assert s3fs.exists("mobysound.org-mirror-v20210704/workshops/5th_DCL_data_bottlenose.zip")
try:
  s3fs.getinfo("5th_DCL_data_bottlenose.zip") # raises exception
  assert False
except Exception as e:
  assert True
  print(f"Got exception: {e}") # Got exception: resource '5th_DCL_data_bottlenose.zip' not found

s3fs = S3FS('dolphicom', dir_path="mobysound.org-mirror-v20210704/workshops/")
assert s3fs.exists("5th_DCL_data_bottlenose.zip")
s3fs.getinfo("5th_DCL_data_bottlenose.zip") # does not raise exception
assert True
```

As a last note, I would suggest to add `s3` to the issue title so that others facing the same issue could find this and to close this issue as well as the PR because I don't think that `s3fs` will change this behaviour.
comment
created
poudro
username_0
The issue I mention only happens when a different user uploads the file whether it be via s3fs or aws cli even when all permitions are ok. 

If the same user as the one trying ot read it via s2fs is the one that did the upload, via either s3fs or aws cli, it works fine. 

So your explanation doesn't hold since it does in fact work, but only sometimes, depending on who did the upload with aws cli.

I'm pretty sure this is not intended behavior on s3fs part as sometimes it works sometimes it doesn't depending on who uploaded the file even when the permitions are ok.
comment
created
geoffjukes
username_2
@username_0 - The explanation does hold. Let me explain it differently....

In a POSIX filesystem, the full file path always exists. So you can 'stat' every part of it. So with "/temp/file.ext" as an example; `temp` exists, and so does `file.ext`

In Object storage (such as S3) it is possible for `/temp/file.ext` to exist WITHOUT `/temp/` existing - because "folders" aren't a thing. Objects use keys that *look* like folder paths, but are not. For `/temp/` to "exist" it has to be created. Quite literally, and empty object with the key `/temp/`.

All that said - I have the same issue (and gripe) with S3FS, because not all S3 clients create the `/foldername/` objects, and I don't always have control over the bucket. If I glob or walk a bucket with this issue, I get errors, and don't have any good way to solve it.

I wish there was a way to tell S3FS to not 'stat' dirs for specific mounts....
comment
created
geoffjukes
username_2
@username_0 Are you saying that files uploaded by 2 users, with the same key prefix, result in an error? e.g. /some/place/user1.txt vs /some/place/user2.txt

Or are the key prefixes different? e.g. /user1/file.ext vs /user2/file.ext

Are the users using the same software to upload/create the objects? Or different software? I'd like to try and recreate your issue, so some specifics would help me to do that.
comment
created
poudro
username_0
@username_2 here's the more detailed scenario:

- There is a single file `foo` that contains `bar`
- There are two users U1 and U2
- U1 uploads the file to `s3://bucket/path/foo` (via s3fs, aws webinterface or aws cli, all three result in same outcome) and sets read permissions so both U1 and U2 can read it

In this scenario U1 can read file via s3fs, but U2 will raise a `ResourceNotFound` even though U2 can successfully access file via aws webinterface and aws cli.
comment
created
geoffjukes
username_2
@username_0 `s3://bucket/path/foo` does not imply that an object the object `s3://bucket/path/` exists, which is what causes the error you are seeing.

When U1 creates the object at `s3://bucket/path/foo` are they first creating `s3://bucket/path` (such as with `makedirs`)? or just creating `s3://bucket/path/foo` directly? I know that the AWS web interface and the AWS CLI allow for direct object creation, without creating the intermediate objects for "faking" the directories - which, again, is the source of the issue you (and I) are experiencing.

Note that "seeing" a "directory" in the web interface, does not imply an object exists with that key.
comment
created
poudro
username_0
The permissions in my experiment were actually set at the bucket level, seems unlikely to me then that it could be linked to permissions at the path level.
comment
created
geoffjukes
username_2
It happens. I've experienced it myself many times.
comment