danilop/yas3fs — pull request #60 by EricByers (2014-10-29 20:46:01+00:00)
Title: Allow 3rd Parameter "Path" to "Reset" Message

First of all thanks for yas3fs, appreciate the work you have done on it.

In our usage of yas3fs we routinely are pushing ~20,000 small files in a matter of minutes (distributed write). Yas3fs keeps up for the most part, but the SNS/SQS handling does not keep up for us, often times lagging 3-4x longer than the initial writes took (until the last message is processed on a different server).

Our setup is that we want to write once in a centralized place with multiple 'readers' that need to stay up to date (and quickly). So we're switching to a style where we upload directly to the s3 bucket, however as it currently stands we would need to send a full reset (which is ok in some cases, but not in a lot of cases) or invalidate every path using "upload".

This pull request adds an optional 3rd parameter to the reset which is a path, and basically looks for any key that exists that starts with the path, and invalidates it. I'm not sure this is correct, since there is also an invalidate_cache (which doesn't seem to invalidate directories). The other thing that may be an issue is when there are lots of keys in cache, and lots of keys that match...could potentially cause performance issues (for instance we loaded up almost everything into cache and then reset a low level path and it was ~45 seconds for ~20k entries, in this case it would probably be better to use a "reset" without a path of course.

Was looking for some additional thoughts/feedback on this approach, if it is acceptable I can revert all the line ending changes my IDE automatically did so it merges in smoother.

Again, thanks for all your work!