phdeniel/nfs-ganesha — Issue #30: cache inode doesnt keep correct number of	files	open
Author: eitanb
Created: 2011-12-15 17:53:57+00:00
State: open

I'm trying to configure Ganesha's cache-inode with a scale of 100K open files, specifically i used 16 workers with cache_content_client_parameter_t::max_fd_per_thread = 10000.
The above configuration is supposed to allow the cache-inode to maintain 16 * 10K = 160 K files open.
However, i see (through /proc/ganesha-pid-#/fds/) that the number of open files increases until it reaches 10K & then remains at that level.
Looking at the code for this issue, i see wrong interpretation of "cache_content_client_parameter_t::max_fd_per_thread" in cache_inode_open() - this function opens a file & then checks its file-descriptor value (using FSAL_FILENO(...) which is actually GPFSFSAL_GetFileno(...)) - it then checks if the fd value is greater than "cache_content_client_parameter_t::max_fd_per_thread" & when true, it activates GC (using cache_inode_gc_fd(..)) to free older entries.
The end result is that the "cache_content_client_parameter_t::max_fd_per_thread" parameter which is supposed to be per-worker-thread is actually enforced globally for the whole process. so no matter how many worker threads you configure, the CacheInode will not hold more than that value of open fd's.

rethinking the design decision
i'm trying to think why is there a need for managing the inode entries per worker. i fully understand the repetitive design point of separating memory pools between threads to reduce contention. but i'm not sure this is the solution here.
assuming the system warms up for some time & that incoming RPC's are spread across all workers, we'll eventually (assuming fixing the logic for correct enforcement of max_fd_per_thread) have exactly that number of entries in cache-inode per worker. from that point on we'll only reuse existing entries which are already allocated.
the only option for shrinking the cache-inode is the GC mechanism but i'm assuming that a high-end server will turn it off because configuring the cache inode to drop entries after a fixed interval doesnt seem to give much, especially with the new Up calls API which allows the file system to notify of changes. if anyone is afraid of having stale information in the cache inode, it should use that kind of file system.

if we go for a global pool then reusing an entry requires taking the oldest object from the list but also to close/flush the open descriptor & open a new file. the filesystem operations is clearly the real chunk of work so grabbing the oldest object off the global list shouldn't become a contention hot spot. and in any case, a high performance server would have to be configured such that a large part of incoming request would be serviced using the cache-inode without referring to the file system at all. otherwise, we'll choke on the file system calls.