meskyanichi/backup — Issue #290: S3 Syncer uploads files which are already stored
Author: fschwahn
Created: 2012-03-01 16:19:08+00:00
State: open

Hi,
unfortunately I don't have a lot of time to investigate this issue, but I found a workaround. The issue is that the S3 syncer uploads files even if they are already on S3. I think this is connected to this fog issue: https://github.com/fog/fog/issues/596 .
The problem is that [bucket.files.to_a](https://github.com/meskyanichi/backup/blob/master/lib/backup/syncer/cloud.rb#L121) returns (sometimes?) a stale list of files from s3. I fixed the issue with quickly with the following code, I'm sure it can be done nicer:

```ruby
def remote_files
@remote_files ||= get_remote_files
end

def get_remote_files
bucket_files = []
bucket.files.each {|f| bucket_files << f}
bucket_files.select { |file|
file.key[%r{^#{remote_base}/}]
}.inject({}) { |hash, file|
key = file.key.gsub(/^#{remote_base}\//,
"#{directory.split('/').last}/")
hash[key] = file
hash
}
end
```

This probably affects the cloudfiles syncer as well.

Best regards,
Fabian