amplab/tachyon — pull request #212 by aarondav (2014-06-09 01:50:52+00:00)
Title: Introduce fine-grained, hierarchical pinning

When a file is pinned, Tachyon will not evict it -- effectively, this allows Tachyon to provide two-level priority caching. The only way to configure the files that should be pinned, however, is via the "pin list", which is determined by system properties that are set when the Tachyon master is started, and cannot be changed at runtime or at a finer granularity than a directory.

This PR introduces a per-Inode concept of "PinConfig", which has 3 states: PINNED, UNPINNED, and DEFAULT. Both files and folders have PinConfig values, though only files can actually be pinned in memory. In order to determine if a file should be pinned, its PinConfig value is first read. If it's DEFAULT, then we will recursively examine the Inode's parents until we find one that has a non-DEFAULT value, or else default to unpinned.

This behavior allows users to dynamically set a folder to be PINNED, for instance, and then any files within that folder are also pinned, unless they are specifically set to UNPINNED. Changing the value of a folder's PinConfig, however, may cause us to traverse all of the folder's children in order to update their pin-ness.

A few implementation notes:
- Compatibility with pinlist was maintained, though it should be considered deprecated.
- TachyonFS.unpinFile() now just sets the Inode to PinConfig.UNPINNED. Since PinConfig overrides pinlist behavior, this maintains behavioral backwards compatibility.
- Inode.mPin and MasterInfo.mFileIdPinList were reused, but with the updated semantics.

Outstanding issues:
- This change does not provide backwards compatibility for EditLog, as I had to add a new field. We should have a more general solution that logs the Tachyon version in the EditLog so we can provide backwards compatibility, or else provide a script to upgrade old journals.
- Use of cached ClientFileInfos can lead to unexpected behavior, because we have no way of pushing the fact that a file was either pinned or unpinned to the users. This only impacts the caching of files, not the eviction, as the latter always re-fetches the mFileIdPinList.