NFSv4: Cache NFSv4 open stateids for the lease period after close

NFSv4 rfc states that -

 --->
 9.10.1. Close and Retention of State Information

 Since a CLOSE operation requests deallocation of a stateid, dealing
 with retransmission of the CLOSE, may pose special difficulties,
 since the state information, which normally would be used to
 determine the state of the open file being designated, might be
 deallocated, resulting in an NFS4ERR_BAD_STATEID error.
 Servers may deal with this problem in a number of ways. To provide
 the greatest degree assurance that the protocol is being used
 properly, a server should, rather than deallocate the stateid, mark
 it as close-pending, and retain the stateid with this status, until
 later deallocation. In this way, a retransmitted CLOSE can be
 recognized since the stateid points to state information with this
 distinctive status, so that it can be handled without error.
 When adopting this strategy, a server should retain the state
 information until the earliest of:

 o Another validly sequenced request for the same open-owner, that is not a retransmission.

 o The time that an open-owner is freed by the server due to period with no activity.

 o All locks for the client are freed as a result of a SETCLIENTID.
 <---

 So, to provide greater assurance and detect replayed CLOSE ops, we chose
 to extend stateid lifetime. Also retaining state_ids extend the life of
 open_owner as well. This will help us in reusing the same open_owner and avoid
 open_confirmation incase of reOPEN ops issued by the client within the lease period.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Additional work done by Frank Filz:

- Resolved merge conflicts

- Dealt with refcounting of state_t objects and need to use
  get_state_entry_ref to access state_entry.

- rename last_close to state_last_close

- Instead of introducing a new state_type, just use state_last_close being
  non-zero to indicate a CLOSE_PENDING state.

- Made significant changes to reap_expired_state_ids since we have to take
  a state_t ref after dropping the hash lock. We also need to get the cache
  inode LRU ref after dropping the hash lock.

- Remove special handling for close in nfs4_Check_Stateid since we no longer
  need it since the stateid hangs around long enough for replay detection.

- Use new atomic store and fetch for time_t to access state_last_close.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
Add LogStateOwner macro to make it easy to log a state_owner_t

Use it most places where NFS v4 state owners are discovered.

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
MORE NFSv4: Cache NFSv4 open stateids for the lease period after close

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
MORE Convert rest of SAL display functions to use display_buffer

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
MORE Use display_buffer for clientid related string creation

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
MORE Make sure client id mutexes are destroyed

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
MORE Add safety to struct state_t

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
Make dump_all_states and dump_all_owners LogFullDebug instead of LogDebug

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
Fix log message in Protocols/NFS/nfs4_op_exchange_id.c to not have \n

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
Make "Expired State, Possibly extra references" message LogFatal

It is much easier to debug this when Ganesha stops immediately...

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
MORE Make sure state_nfs4_owner_unlock_all handles stale cache entry and export

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
MORE Make sure release_openstate handles stale cache entry and export

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
MORE Add LogStateOwner macro to make it easy to log a state_owner_t

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>
MORE NFSv4: Cache NFSv4 open stateids for the lease period after close

Signed-off-by: Frank S. Filz <ffilzlnx@mindspring.com>