ceph/s3-tests — pull request #26 by alfredodeza (2014-12-12 19:35:52+00:00)
Title: fixes/improves weird hashing issues on readwrite tests

There is a bit of repetition in these new additions (a function and a class) but I didn't want to go ahead and refactor anything else unless it fixed the problems we were seeing in http://tracker.ceph.com/issues/10066

Adds some unit tests for the new helpers, and tested these changes against a magna box:

test yaml:
$ cat test.yml
readwrite:
bucket: rwtest
readers: 10
writers: 3
duration: 300
files:
num: 10
size: 2000
stddev: 500
random_seed:
writer: 236047946
main: 2514311331
names: 4153910459
contents: 1256193726
reader: 2312633921
s3:
user_id: client0-system-user
access_key: secret
secret_key: secret
display_name: Mr. foo.client.0
email: adeza@redhat.com
host: magna077.ceph.redhat.com
is_secure: false
port: 8000
port: 7280

And the tests:

cat test.yml | s3tests-test-readwrite
/home/adeza/.virtualenvs/test/local/lib/python2.7/site-packages/pkg_resources.py:991: UserWarning: /home/adeza/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
warnings.warn(msg, UserWarning)
Using random seeds: {'writer': 236047946, 'main': 2514311331, 'names': 4153910459, 'contents': 1256193726, 'reader': 2312633921}
Created bucket: rwtest
Creating random file names
Uploading initial set of 10 files
Starting main worker loop.
Using file size: 2000 +- 500
Spawning 3 writers and 10 readers...

{bucket: rwtest, duration: 220690966, key: ztovozlqkxhqlkt, start: 1418410923.451777,
type: r, worker: 2}
{bucket: rwtest, duration: 89059114, key: ilzimnupha, start: 1418410923.632474, type: r,
worker: 0}
{bucket: rwtest, duration: 86205006, key: bsligrqsqpokbmgek, start: 1418410923.642961,
type: r, worker: 5}
{bucket: rwtest, duration: 285329819, key: rqelwjfddyoscw, start: 1418410923.479829,
type: r, worker: 1}
{bucket: rwtest, duration: 211473942, key: xsvleodvmtbggym, start: 1418410923.573156,
type: r, worker: 8}
{bucket: rwtest, duration: 71174145, key: yurlswmludjlot, start: 1418410923.735761,
type: r, worker: 5}
{bucket: rwtest, duration: 188899994, key: bsligrqsqpokbmgek, start: 1418410923.628557,
type: r, worker: 7}
{bucket: rwtest, duration: 1334789991, key: rqelwjfddyoscw, start: 1418410922.493062,
type: r, worker: 9}
{bucket: rwtest, duration: 272337914, key: yurlswmludjlot, start: 1418410923.56586,
type: r, worker: 4}
{bucket: rwtest, duration: 143549919, key: yurlswmludjlot, start: 1418410923.725403,
type: r, worker: 0}

.....
{bucket: rwtest, duration: 233824968, key: xsvleodvmtbggym, start: 1418410923.675111,
type: r, worker: 2}
Cleaning bucket key
Cleaning bucket key
Cleaning bucket key
Cleaning bucket key
Cleaning bucket key
Cleaning bucket key
Cleaning bucket key
Cleaning bucket key
Cleaning bucket key
Cleaning bucket key