FileStore: Fix _do_sparse_copy_range don't truncate zero sections

If calling clone_range and zero filled sections is at the ending of src
range, the target fd won't write zero filled sections. If target's
origin size shorter than needed, it will cause inconsistence between
two fd's desired range.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
StoreTest: Add tests for clone_range op

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Messenger: Add unit tests

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
PipeConnection: Modify "is_connected" semantic

"is_connected" should be used to judge whether have built a successful
connection with peer. Since "pipe == NULL" is useless for whether connected
because pipe can be construct without any potential network problem

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
FileStore: Move ftruncate to do_sparse_copy

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
AsyncConnection: Avoid hungry if mark_down's caller is the eventcenter's owner

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
AsyncConnection: Avoid name conflicting with AsyncMessenger

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
AsyncMessenger: Fix accept connection replacing process

Original there exists bugs that we don't modify existing file event,
it will let remaining alive connection's event won't be called by
EventCenter.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
AsyncConnection: Add loopback connection support

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
WBThrottle: make bytes/ios/inode_wb's perf counter effective

Since sync thread will cause unstable iops and latency performance curve, we
may want make WBThread do more(or moderate?) writeback and avoid sync thread
flush too much which will cause journal io long tail.

Via these counters, we can view how much object or bytes are write backed by
WBThread and how much bytes are flushed . Then we can have a great tuning for
"*bytes_start_flusher", "*ios_start_flusher" and "*inodes_start_flusher".
What we want to see is that in-memory data can writeback into disk with a
moderate rate.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>
Include common/likely.h in buffer.h

Signed-off-by: Matt Benjamin <matt@cohortfs.com>
os: IndexManager:: optimaze map<coll_t, CollectionIndex* > col_indices
col_indices is just used to cache CollectionIndex, and use col_indices.find() to get its corresponding index. Without any using of up_bound() and low_bound(), we can use unordered_map to make it fast. Based on perf, when I chanage map to unordered_map, the cpu cost for get_index(coll_t c, const string& baseDir, Index *index) is much lower.

Signed-off-by: Ning Yao <zay11022@gmail.com>
OSD: FileJournal: call writeq_cond.Signal if necessary in submit_entry

Signed-off-by: Xinze Chi <xmdxcxz@gmail.com>
Add safe-sharing to buffer::list and buffer::ptr.

Formerly known as "strong claim," this change introduces a new
is_shareable() property to ceph::buffer::raw objects, which can be
used to prevent unplanned sharing of buffers.

The default value of is_shareable() is true, but derived classes may
override this.  For example, Accelio message buffers may point to
registered or pool-allocated memory which should not be held for
long periods.

Cloning of non-shareable buffers is now the unmarked case, so is
performed automatically in the buffer::list copy constructor.
Low-level operations should allow explicit volatile sharing,
so copying a buffer::list:iterator or a buffer::ptr works as normal.

For explicit sharing of "non-shareable" buffers, a new boolean is added to
the public claim_* methods, and a buffer::list::share method is also
added, to share an entire sequence.

Signed-off-by: Vu Pham <vu@mellanox.com>
Signed-off-by: Matt Benjamin <matt@cohortfs.com>
Restore zero-copy buffers in OSD fast path.

This change restores volatile sharing semantics in the Message
decode path, and also in the OSD write path for FileStore/FileJournal.

This can be verified with a breakpoint set at the clone/COW case in
buffer::ptr::make_shareable(currently buffer.cc:690).

Signed-off-by: Matt Benjamin <matt@cohortfs.com>
Add diagnostic unshareable buffer.

Signed-off-by: Matt Benjamin <matt@cohortfs.com>
Add unittests for safe sharing and make_shareable()

Checks the following assertions:

1. a bufferlist shared with buffer::list::share yields shared
raw memory

2. a bufferlist copied with buffer::list::list(const list&) yields
shared raw memory

3. an unshareable bufferlist (bufferlist with unshareable segments)
copied with buffer::list::list(const list&) yields unshared raw
memory

4. an unshareable bufferlist claimed with CLAIM_ALLOW_NONSHAREABLE
yields shared raw memory

Signed-off-by: Matt Benjamin <matt@cohortfs.com>
ReplicatedPG: Don't use the return value of getattr_maybe_cache as attr length.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
ReplicatedPG: For MAPEXT, using bl.length() instead of op.extent.length as num_rd_kb.

For fiemap, it don't real read the data(offset,length). It only fetech the infos of
extents. So don't use length as num_rd_kb.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
os: free space tracking for MemStore

Allow users to set an artificial upper bound
on size of the memstore OSD, and report actual
usage information in fsstat.

This is a useful tool for simulating nearly-full
systems.

Signed-off-by: John Spray <john.spray@redhat.com>