Persistent offsets in dirent AVL

Make offsets (ie, cookies) persistent.  Any cookie allocated since the
full population of a dirent cache is preserved, the associated dirent is
marked deleted.  If a deleted offset is ever hashed later, the slot is reused.

The strategy is to move deleted entries to an alternate AVL.  If a READDIR
request is made for an offset which is no longer active, entries are returned
starting from the least upper bound of offset in the -active- AVL.  Unit
tests for the new avltree_sup and former avltree_inf methods are added.

A hard upper bound of 64K deleted entries per cache entry is enforced.  Further,
we expect that deleted entries may be purged after periods of inactivity.
(Persistent cookies are a convience for broken clients.)
Change BUDDY_ADDR_T to void *

BUDDY_ADDR_T was originally defined to expand to caddr_t.  This has
the unfortunate effect that the pointer returned from Mem_Alloc must
be explicitly cast to be assigned to any pointer variable not of type
char * or caddr_t.  This has the effect of cluttering the code and
reducing type safety (if, for example, a function allocating objects
of a specific type were added somewhere, old code assigning its
pointers to something inappropriate would receive no warning because
of the cast.)
HashTable baseline and basic improvements

This is preliminary to the work to the Hash Table refactor.  It fixes
bugs and infelicities in the current Hash Table interface before the
implementation of latching hash table operations and better memory
handling.  Improvements include:

  * Use of void* rather than caddr_t in the hash table buffer
    descriptors.  This removes the need to litter the code with casts
    (and the attendant typing problems.)

  * Update the documentation to match the code in HashTable.c.

  * Fix a few lock and memory leaks.

  * Make the index into the array of Red-Black trees a uint32_t and
    the hash in a red-black tree a uint64_t.  This allows one to hash
    things like clientids and pointers using a modulus for one and the
    identity function for the other.

  * Use POSIX read-write locks.  There was talk of doing using them on
    the mailing list, and, more importantly, they have a single
    'unlock' operation rather than an 'unlock-read' or 'unlock-write'
    operation.  This is important for the 'latched' hash access to
    come.
HashTable: Rewrite/Refactor

This patch accomplishes two goals.  First, it provides an
implementation of the *Ex series of hashtable calls as requested,
though the names are changed:

 * HashTable_GetLatch: This function looks up a key in a hash table
   and, on success and not-found, retains the lock on the hash
   partition.
 * HashTable_ReleaseLatched: This function releases a lock acquired in
   HashTable_GetLatch.
 * HashTable_SetLatched: This function sets a value in the hash table,
   the key must be the same as that supplied to HashTable_GetLatch.
 * HashTable_DelLatched: This function removes a value from the hash
   table.  The key supplied must be the same as that supplied to a
   previous call of HashTable_GetLatch.

Both HashTable_SetLatched and HashTable_DelLatched release the lock
held on the table.  See the Doxygen comments in HashTable.c for more
information.

HashTableGetLatch fills (and the -Latched functions consume) a datum
of type struct hash_latch.  It contains the index (identifying the
partition) and hash (identifying the place in the tree) values
computed during lookup and the node found (or NULL if no node was
found) so that the values need not be recomputed on later calls.  I
was asked about getting index value for a key, and it can be retrieved
from here.

The Set, Get, Del, GetRef, DelRef, and Test_And_Set functions were
implemented in terms of the above primitives.

The second goal was to cut down the number of allocations used in a
hash table and to give it better cache behaviour by merging the arrays
together into a single one of type struct hash_partition.
Make HashTable_Init take a pointer

There is no reason to copy the hash parameters to the stack so that
HashTable_Init can copy them into the hash table structure.  Instead,
just pass a pointer.
Move declaration of ht_client_id to nfs_reaper_thread.c
Revert "Move declaration of ht_client_id to nfs_reaper_thread.c"

Frank Filz is going to handle this as part of the clientid changes.
Remove default case from hash_table_err_to_str

As was pointed out by Frank Filz, the default case prevents the
compiler from issuing a warning if a error code is added without a
matching string.  Therefore, get rid of the default and just return
the UNKNOWN string on fall-through.
Fix compile failure of SNMP thread
Fix HashTable_DelRef

Do not pass garbage to dereference procedure on HASHTABLE_NOTFOUND.
This seems to fix the Connectathon lock test failure.
Don't return spurious success from HashTable_Del
Strict compile and rebase fixes
Remove skip-deleted logic (no longer needed).  Clear dirent flags after GetFromPool.
Fix error in HashTable_Get_and_Del reported by Rong Zeng

In the original code, HashTable_Get_and_Del would attempt to delete an
entry even if none was found by the original GetLatch.  Instead we
release the latch and return the appropriate code.
Fix strict compile failures
make nfsv4 recovery/stable storage better
fix problem which failed pynfs testcase RNM13 - mv -T dir1 empty_dir2
fix problem which failed pynfs testcase CMT4: check commit range is over limit
Removed Use_OPEN_CONFIRM from VFS default config.
This parameter no longer exists.
Also removed parameters that pertain to logging.
Put an ifdef _CONFORM_TO_TEST_LOCK8c around new lock seqid !=0 check.
Per the  RFC 3530b, The server is free to assign any value for the first
request issued for any given state-owner. But the pynfs lock8c test
expects this kind of behavior.

Signed-off-by: Venkateswararao Jujjuri "<jvrao@linux.vnet.ibm.com>