ceph/ceph — pull request #2807 by thesues (2014-10-27 08:24:00+00:00)
Title: Set the default RadosSTriperImpl refcnt from 0 to 1

Dear list,

I have met this when I was using radosstriper C API. My program is
roughly like this:

1. rados_striper_aio_write
2 rados_aio_flush
3. rados_aio_wait_for_safe
4. rados_aio_release
5. rados_striper_destroy
6. rados_ioctx_destroy
7. rados_shutdown

In most time, this works well, But the programm occasionally
hangs forever. Output of gstack:

Thread 1 (Thread 0x7fe0afba0760 (LWP 18509)):
0 0x000000330f20822d in pthread_join () from /lib64/libpthread.so.0
1 0x000000347566cea2 in Thread::join(void**) () from /usr/lib64/librados.so.2
2 0x00000034755ac535 in librados::RadosClient::shutdown() () from /usr/lib64/librados.so.2
3 0x0000003475592269 in rados_shutdown () from /usr/lib64/librados.so.2
4 0x0000000000402349 in main ()

Thread 4 (Thread 0x7fe0ab14d700 (LWP 18541)):
0 0x000000330f20e264 in __lll_lock_wait () from /lib64/libpthread.so.0
1 0x000000330f209508 in _L_lock_854 () from /lib64/libpthread.so.0
2 0x000000330f2093d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
3 0x0000003475633af1 in Mutex::Lock(bool) () from /usr/lib64/librados.so.2
4 0x00000034755abd37 in librados::RadosClient::put() () from /usr/lib64/librados.so.2
5 0x0000003475592501 in librados::Rados::shutdown() () from /usr/lib64/librados.so.2
6 0x00007fe0afbba9f7 in libradosstriper::RadosStriperImpl::CompletionData::~CompletionData() () from /usr/lib64/libradosstriper.so.1
7 0x00007fe0afbbaad9 in libradosstriper::RadosStriperImpl::WriteCompletionData::~WriteCompletionData() () from /usr/lib64/libradosstriper.so.1
8 0x00007fe0afbc1d75 in RefCountedObject::put() () from /usr/lib64/libradosstriper.so.1
9 0x00007fe0afbc224d in libradosstriper::MultiAioCompletionImpl::safe_request(long) () from /usr/lib64/libradosstriper.so.1
10 0x00000034755c5ce8 in librados::C_AioSafe::finish(int) () from /usr/lib64/librados.so.2
11 0x00000034755a0e89 in Context::complete(int) () from /usr/lib64/librados.so.2
12 0x000000347564d4c8 in Finisher::finisher_thread_entry() () from /usr/lib64/librados.so.2
13 0x000000330f2079d1 in start_thread () from /lib64/libpthread.so.0
14 0x000000330eae886d in clone () from /lib64/libc.so.6

When the last CompletionData were released, it will release
RadosStriperImpl itself and the RadosClient. There are two shutdown threads in the same
time. I think since API caller will always call rados_striper_destroy to
release memory. we do not need the RadosStriperImpl to be released
automacally, and it will also have race conditions like this one;

any thoughts?