mds: make sure table request id unique

When a MDS becomes active, the table server re-sends 'agree' messages
for old prepared request. If the recoverd MDS starts a new table request
at the same time, The new request's ID can happen to be the same as old
prepared request's ID, because current table client assigns request ID
from zero after MDS restarts.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: send table request when peer is in proper state.

Table client/server should send request/reply when the peer is active.
Anchor query is an exception, because MDS in rejoin stage may need
fetch files before sending rejoin ack, the anchor server can also be
in rejoin stage.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: make table client/server tolerate duplicated message

Anchor client re-sends queries when the anchor server becomes active.
So it's possible to get duplicated query reply.

When the table server recovers, the clients re-send commits to the
server, the server re-sends 'agree' messages to the clients. When
the clients receive the 'agree' messages, they may send another
commit/rollback message to the server.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: mark connection down when MDS fails

So if the MDS restarts and uses the same address, it does not get
old messages.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: consider MDS as recovered when it reaches clientreply state.

MDS in clientreply state already start servering requests. It also
make MDS::handle_mds_recovery() and MDS::recovery_done() match.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: defer eval gather locks when removing replica

Locks' states should not change between composing the cache rejoin ack
messages and sending the message. If Locker::eval_gather() is called
in MDCache::{inode,dentry}_remove_replica(), it may wake requests and
change locks' states.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: unify slave request waiting

When requesting remote xlock or remote wrlock, the master request is
put into lock object's REMOTEXLOCK waiting queue. The problem is that
remote wrlock's target can be different from lock's auth MDS. When
the lock's auth MDS recovers, MDCache::handle_mds_recovery() may wake
incorrect request. So just unify slave request waiting, dispatch the
master request when receiving slave request reply.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: don't delay processing replica buffer in slave request

Replicated objects need to be added into the cache immediately

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: compose and send resolve messages in batch

Resolve messages for all MDS are the same, so we can compose and
send them in batch.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: don't send resolve message between active MDS

When MDS cluster is resolving, current behavior is sending subtree resolve
message to all other MDS and waiting for all other MDS' resolve message.
The problem is that active MDS can have diffent subtree map due to rename.
Besides gathering active MDS's resolve messages are also racy. The only
function for these messages is disambiguate other MDS' import. We can
replace it by import finish notification.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: set resolve/rejoin gather MDS set in advance

For active MDS, it may receive resolve/resolve message before receiving
the mdsmap message that claims the MDS cluster is in resolving/rejoning
state. So instead of set the gather MDS set when receiving the mdsmap.
set them in advance when detecting MDS' failure.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: don't send MDentry{Link,Unlink} before receiving cache rejoin

The active MDS calls MDCache::rejoin_scour_survivor_replicas() when it
receives the cache rejoin message. The function will remove the objects
replicated by MDentry{Link,Unlink} from replica map.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: send cache rejoin messages after gathering all resolves

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: send resolve acks after master updates are safely logged

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: fix MDS recovery involving cross authority rename

For mds cluster, rename operation may involve multiple MDS. If the
rename source's auth MDS crashes after some witness MDS have prepared
the rename but before the rename is committing. Later when the MDS
recovers, its subtree map and linkages are different from the prepared
MDS'. This causes problems for both subtree resolve and cache rejoin.
The solution is, if the rename source's auth MDS fails, the prepared
witness MDS query the master MDS if the operation is committing. If
it's not, rollback the rename, then send resolve message to the
recovering MDS.

Another similar case is a prepared witness MDS crashes when the
rename source's auth MDS has prepared or is preparing the operation.
when the witness recovers, the master just delay sending the resolve
ack message until the it commits the operation.

This patch also updates Server::handle_client_rename(). Make preparing
the rename source's auth MDS be the final step before committing the
rename.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: remove MDCache::rejoin_fetch_dirfrags()

In commit 77946dcdae (mds: fetch missing inodes from disk), I introduced
MDCache::rejoin_fetch_dirfrags(). But it basicly duplicates the function
of MDCache::open_undef_dirfrags(), so just remove rejoin_fetch_dirfrags()
and make open_undef_dirfrags() also handle undefined inodes.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: include replica nonce in MMDSCacheRejoin::inode_strong

So the recovering MDS can properly handle cache expire messages.
Also increase the nonce value when sending the cache rejoin acks.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: encode dirfrag base in cache rejoin ack

Cache rejoin ack message already encodes inode base, make it also encode
dirfrag base. This allowes the message to replicate stray dentries like
MDentryUnlink message. The function will be used by later patch.

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: handle linkage mismatch during cache rejoin

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>
mds: reqid for rejoinning authpin/wrlock need to be list

Signed-off-by: Yan, Zheng <zheng.z.yan@intel.com>