LU-2182 utils: new llapi_layout API implementation

Add a new set of llapi routines for interacting with file layouts that
hide the details of the wire-protocol data structures from the user.
Define an opaque data type lustre_layout_t to abstract the layout of a
lustre file and generic accessor functions to read and write it.  The
following functions are added the liblustreapi public interface:

  llapi_layout_alloc - allocate a new layout
  llapi_layout_file_create - create new file with given layout
  llapi_layout_free - free memory allocated for a layout
  llapi_layout_lookup_bypath - get file layout given a path
  llapi_layout_lookup_byfd - get file layout given a file descriptor
  llapi_layout_lookup_byfid - get file layout given a file identifier
  llapi_layout_obj_id - get object id associated with a stripe
  llapi_layout_ost_index - get OST index associated with a stripe
  llapi_layout_ost_index_set - set OST index assoicated with a stripe
  llapi_layout_pattern - get RAID pattern of a layout
  llapi_layout_pattern_set - set RAID pattern of a layout
  llapi_layout_pool_name - get pool name of a layout
  llapi_layout_pool_name_set - set pool name of a layout
  llapi_layout_stripe_count - get stripe count of a layout
  llapi_layout_stripe_count_set - set stripe count of a layout
  llapi_layout_stripe_size - get stripe size of a layout
  llapi_layout_stripe_size_set - set stripe size of a layout

The layouts are read and written using fgetxattr() and fsetxattr()
instead of ioctl() to make it easier for architectures like Blue Gene
to function-ship the system calls.

Signed-off-by: Ned Bass <bass6@llnl.gov>
Change-Id: I35fb51055b6438ef3090f43c28a4083a66eaa907
Revert "LU-2948 osc: Add timeout to osc_lru_reserve"

This reverts commit 1bf5f1c02b51d42a326c93bf1a285cd749de4ae6.
Revert "LU-2139 osc: Add OBD_BRW_SOFT_SYNC page flag"

This reverts commit c9052719db9c1b3857a142e278e53cd98c0170ae.
Revert "LU-2139 osc: Track number of "unstable" pages per osc"

This reverts commit 8e2481383224f94ee120e8ab28d06196e045ea43.
Revert "LU-2139 osc: Track number of "unstable" pages on client"

This reverts commit 29701a0535334a835d711f4467e9fd8459a3fea3.
LU-2948 osc: Add timeout to osc_lru_reserve

If a thread ends up sleeping on the osc_lru_waitq, this patch ensures it
is woken up at least every 60 seconds and tries to perform LRU
reclaiming again. When the time out hits, a message is printed to the
console stating this happened. In addition, a lustre debug log is dumped
with D_CACHE information to help debug why the thread failed to reclaim
in the first place and why it was never woken up.

Test-Parameters: fortestonly
Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Iac643f2abb424f523d32265f93b4d05c828b0649
Patch-Set: 1
LU-2139 obdclass: Rename max_dirty_pages variable

This patch renames the "max_dirty_pages" global variable to
"max_pinned_pages". This is in anticipation for tracking the number of
outstanding unstable pages on the system, and limiting the sum of the
outstanding dirty and unstable pages by this value. Thus, the previous
name doesn't fit since it will be used to limit both dirty and unstable
pages. The mnemonic being both dirty and unstable pages are "pinned" in
memory, so this is the maximum number of lustre pages pinned in memory
at any point in time.

As a result of the change described above, the file serving as the
interface to setting and getting this value was change as well.
Previously it was named "max_dirty_mb", now it is named "max_pinned_mb".

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ibed3865a80ef153855e9ca4b570837ac24a4df23
Patch-Set: 4

Conflicts:

	lustre/osc/osc_cache.c
LU-2139 osc: Track and limit "unstable" pages

This change adds a global counter to track the number of "unstable"
pages held by a given client, along with per file system counters. An
"unstable" page is defined as a page which has been sent to the server
as part of a bulk request, but is uncommitted to stable storage.

In addition to simply tracking the unstable pages, they now also count
towards the maximum number of "pinned" pages on the system at any given
time. Thus, a client will now be bound on the number of dirty and
unstable pages it can pin in memory. Previously only dirty pages were
accounted for in this limit.

In addition to tracking the number of unstable pages in Lustre, the
NR_UNSTABLE_NFS memory zone is also incremented and decremented for
easy monitoring using the "NFS_Unstable:" field in /proc/meminfo.
This field is also used internally by the kernel to limit the total
amount of unstable pages on the system.

The motivation for this change is twofold. First, the client must not
allow itself to disconnect from an OST while still holding unstable
pages. Otherwise, these unstable pages can get lost due to an OST
failure, and replay is not possible due to the disconnect via unmount.

Secondly, the client needs a mechanism to prevent it from allocating too
much of its available RAM to unreclaimable pages pinned by the ptlrpc
layer. If this case occurs, out of memory events can trigger as a side
effect, which we need to avoid.

The current number of unstable pages accounted for on a per file system
granularity is exported by the unstable_stats proc file, contained under
each file system's llite namespace. An example of retrieving this
information is below:

	$ lctl get_param llite.*.unstable_stats

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ic43d34bfa39da9ac4fe159000c7db5908467fd7b
Patch-Set: 29

Conflicts:

	lustre/osc/osc_cache.c
LU-2139 osc: Track number of "unstable" pages per osc

This change adds simple accounting hooks for "unstable" pages on a per
OSC basis. Now, in addition to the per filesystem tracking, each OSC
will maintain a running total of its unstable pages. These counters are
exported through the proc interface, and can be read using the lctl
command.

For example:

    # Read number of unstable pages contained by each OSC
    lctl get_param osc.*.unstable_stats

The motivation for this change is in anticipation of implementing a
"soft sync" functionality, urging servers to commit these unstable
pages to stable storage. The per OSC accounting allows a client to
limit the soft sync request to only the OSCs which have outstanding
unstable pages.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ife1f90515cb60ba9f85daba5d50483562f2697d1
Patch-Set: 7
LU-2139 osc: Use SOFT_SYNC to urge server commit

This change adds a BRW page flag, OBD_BRW_SOFT_SYNC. This flag is
intended to urge a server to commit a client's unstable pages to
stable storage. A client will add this flag to any BRW requests while
it is in a state where it has "many" unstable pages pinned in its cache.

The server, upon receiving a page with this flag set, *should* begin
an async commit. The idea being that, with the proactive async commit,
the client's pinned unstable pages will transition into a stable state
faster than they would have otherwise. Although, the server side portion
of this agreement is still unimplemented, so the OBD_BRW_SOFT_SYNC flag
will currently fall on deaf ears.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I08946a5ec6afab390655643f6d3c8320bf010aea
Patch-Set: 12
LU-2139 ofd: Do async commit if SOFT_SYNC is seen

This change implements the server side portion of the OBD_BRW_SOFT_SYNC
agreement. When this flag is seen on an incoming BRW page, the server
will begin an asynchronous commit operation via dt_commit_async.

The intention is for this operation to commit pages belonging to a
client which has "too many" outstanding unstable pages in its cache.
Upon the commit operation completing, the last_commited value should
percolate out to the client of interest, allowing it to drop some of
its pinned unstable pages.

Unfortunately, there is currently no optimization to proactively send
the last_committed value to the client immediately after the commit
operation succeeds. It must be piggybacked on any future IO or
pings made by the client.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I1bac4bc517ab023a5b298f34bb695c7ba8ece896
Patch-Set: 1
LU-3103 Fix function declared static and EXPORT_SYMBOL

In LU-2912 commit 7e915f5d7177b22bd3cc800137fb505781a2c037,
the function linkea_entry_pack() was accidentally delared static
and then also explicitly exported with EXPORT_SYMBOL.  On ppc64
gcc balks at this conflict.

Change-Id: I60093fc3da8b82e51530ed93427e5ee8d8e6745d
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Fix build system to support new zfs 0.6.1 packaging

Change-Id: Iac421e845c7cb894abbc36dca4966489d2ad2c65
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
LU-2182 utils: Change include attr/xattr.h to sys/xattr.h

This will likely be merged into the change titled:

  LU-2182 utils: new llapi_layout API implementation

Change-Id: Idc047397adbfe02e5ea52944a6098631bb35f98f
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>