Revert "LU-4707 build: Don’t deploy the "lustre" init script on clients"

This reverts commit 45c228ec90b35c5d6ff295643cf1eeea032f5499.

Under mock-based build systems, this failed to include the init
scripts even for servers.

Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
LU-1199 build: Make "make rpm" use "make src" output

We rework the rpm and srpm Make targets.  The srpm
target generates the source rpm package much as it did
before, but it puts the results in a known temporary
rpm build tree.  This allows the "make rpm" target to
simply depend on the srpm target, and then build the
binary rpms from the source rpm.

Previously the rpm target _also_ generated a source rpm
package, which could be redudant, and failed to verify that
the source rpm package was itself buildable.

Further the patch moves the construction of the rpmbuild
command line options out of the makefile and into the m4
files.  Scripting is much easier in the m4 where the contents
are mostly shell anyway, and it also removes a somewhat
confusing layer of indirection from the top of the
autoMakefile.am file.

Change-Id: Id0612ade9f170d84518f01d4e8de654a7ba8af88
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
LU-1199 build: Pass rpmbuild options to configure

We modify the spec file to have it pass lustre build options
specified on the rpmbuild command line down into the configure
option.  A conversion from rpmbuild option to configure option
happens, with each option stored in its own spec file macro
when needed.

Some of the options that we already present are converted from
just rewriting the CONFIGURE_ARGS environment variable to having
their own explicit rpm macro.  This keeps the settings clear
and disctinct right up until the moment they are used on the
command line to the configure command.

Change-Id: I154d544c7cddef6194819b69d2238497ab477a54
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
LU-1199 build: Build against installed rather than running kernel

By default, the spec file attempts to build Lustre against the kernel
that is currently running on the builder node.  This is not necessarily
the best default, because it does not play with with a build system
that employs the "mock" command.

With mock, we have a chroot environment with a minimal required set
of packages needed to build the rpms.  The kernel that is present
in the chroot environment, and its associated source files, are not
at all necessarily related to the kernel running on the host computer
that is running the mock command.

Therefore, we really want to make the default behavior of Lustre's
build process to look in the right locations for a given Linux
distro to find the newest installed kernel build file.

This patch makes that change.  It only explicitly supports RedHat 6.X,
but implements it in way that allows adding support for other
distributions.

This also adds appropriate a few appropriate Requires and BuildRequires
fields.

Change-Id: Ife4e2cf8458af7428a571b2f53fbdad71d9bd0ca
LLNL-xxx Don't close connection for timedout ZC_REQ

Originally filed as b=23352

Change-Id: Id863c6d9e5180b3dcc8612f6be3e0da9f419c193
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
LU-165 o2iblnd: Make IB privileged ports required by default.

Change-Id: I4564d0507c0d7cce90f957ccfc26dd36ab189ff3
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
LLNL-xxx Tune adaptive timeouts

Simul test 4 causes soft lockups on OSTs llnl bug 761.  Because of
this we need to increase at_min to 15 seconds to avoid this failure
when we are dealing with roughly 10,000 pending locks.  Once this
algorithm is improved we may be able to remove this.

We have also increase at_early_margin to 10 seconds because we do
see what looks like excessive network transit times and this gives
us a bit larger hedge against that.

See bugzilla 18066.

Change-Id: I3128edcf04d44c6e1d95793ae7e72ea872280089
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Jim Garlick <garlick@llnl.gov>
LU-1095 debug: Standardize, suppress mount/umount messages

Standardize mount/umount console message to include profile name,
and optionally suppress them with the 'quiet' mount option.  We
have been using private namespaces for testing and mounting then
umounting the FS as needed for each job.  In this context these
messages end up causing alot of syslog noise.

Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Change-Id: I7514f6016c337a358e5e31146644810dff292d02
LU-1095 debug: Standardize recovery messages

The guiding principle here is that any message which might normally
occur as part of recovery should be printed using LCONSOLE_*.  Also
attempt to limit to a single message per service the final result
of the recovery process.  Too much noise and it will always just
get ignored.

Change-Id: I81deab5c0020573126fdceded3e006727206c5e1
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
LU-1095 debug: Standardize mount/umount message

A first attempt to cleanup some of the gratutious messages which
are observed when a service mount/unmounts (or fails to).  During
normal start up a single message that the service was mounted is
the goal.  On failure a single message describing the failure
should be enough since the returned error isn't always sufficient
to clearly determine the problem.

As always LCONSOLE_* messages are prefered for any error we expect
an administrator to see during normal usage.

Change-Id: I65ba63f40b596e24d0b7cd46b7485a83a10c67be
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
LU-1095 debug: Log expired requests as D_NETERROR

While knowing about expired requests is important, it's also not
an uncommon event on a large system.  As long as the lower layers
can retry or take other corrective action there's no reason to
both an administrator with this.  They can always set D_NETERROR
is they suspect problems with the network.

Change-Id: I0bf06d5df82bbd9fe9e3f223ed6b67de2d5b69b5
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
LU-1353 recovery: debug patch for open replay

Open replay may fail on -ENOENT, add debug message to print fid and
etc.

Signed-off-by: Lai Siyao <laisiyao@whamcloud.com>
Change-Id: Ic7b2e796429934d8996621f648107d11cf9f3373
LU-2327 debug: dump info on request and import

if specific imp_lock can not be acquired in 5s

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: Ie23343cbcfce9a3a027786c07e7d0a42548d2bda
Revision-Id: 1

A small tweak was added to overcome build failures resulting from the
use of kernel only symbols in user space.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ie63b422e3fb80fca932b77d504b3f824626beded
LU-2327 ptlrpc: Print spinlock owner information

If CONFIG_DEBUG_SPINLOCK is enabled, the spinlock owner and magic values
are now printed in the "spinning too long" message.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ib9e13833e5a67d03792edd8eaee519493ba50c9e
LU-2669 scripts: Change behavior of lustre_rmmod

The lustre_rmmod script was modified to take an arbitrary list of
modules and try to remove them and any modules dependent on them.
Previously its behavior was to always remove the libcfs module, along
with either the ldiskfs or another module passed as a parameter.

The old interface was roughly maintained. Any of the following commands
will remove the ldiskfs, libcfs, and all dependent modules:

    $ lustre_rmmod
    $ lustre_rmmod ldiskfs
    $ lustre_rmmod ldiskfs libcfs

The benefit now, is that any other list of modules can be specified
without removing libcfs. For example, the following command will only
remove ptlrpc and its dependent modules (leaving libcfs intact):

    $ lustre_rmmod ptlrpc

This was not previously possible. As a result, the lnet init script was
modified to use this new functionality instead of rolling its own
implementation to obtain the same result.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Id9732f399d1bc7d7d4f1ec4f684fa16cab35e786
Patch-Set: 5
LU-2067 debug: Redirect ldlm_resource_complain() to D_DLMTRACE

ldlm_resrouce_complain() causes a large amount of console messages
that sysamins do not understand, and it requires no intervention.

Silence this message until someone fixes the underlying issue.

Change-Id: I09a0631242579147332a53fcb0c115891490b63f
Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
LU-2139 osc: Track and limit "unstable" pages

This change adds a global counter to track the number of "unstable"
pages held by a given client, along with per file system counters. An
"unstable" page is defined as a page which has been sent to the server
as part of a bulk request, but is uncommitted to stable storage.

In addition to simply tracking the unstable pages, they now also count
towards the maximum number of "pinned" pages on the system at any given
time. Thus, a client will now be bound on the number of dirty and
unstable pages it can pin in memory. Previously only dirty pages were
accounted for in this limit.

In addition to tracking the number of unstable pages in Lustre, the
NR_UNSTABLE_NFS memory zone is also incremented and decremented for
easy monitoring using the "NFS_Unstable:" field in /proc/meminfo.
This field is also used internally by the kernel to limit the total
amount of unstable pages on the system.

The motivation for this change is twofold. First, the client must not
allow itself to disconnect from an OST while still holding unstable
pages. Otherwise, these unstable pages can get lost due to an OST
failure, and replay is not possible due to the disconnect via unmount.

Secondly, the client needs a mechanism to prevent it from allocating too
much of its available RAM to unreclaimable pages pinned by the ptlrpc
layer. If this case occurs, out of memory events can trigger as a side
effect, which we need to avoid.

The current number of unstable pages accounted for on a per file system
granularity is exported by the unstable_stats proc file, contained under
each file system's llite namespace. An example of retrieving this
information is below:

	$ lctl get_param llite.*.unstable_stats

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I67ab9ee244e6d6081069e3c3d5b2e493ddb97874
Patch-Set: 3
LU-2139 osc: Track number of "unstable" pages per osc

This change adds simple accounting hooks for "unstable" pages on a per
OSC basis. Now, in addition to the per filesystem tracking, each OSC
will maintain a running total of its unstable pages. These counters are
exported through the proc interface, and can be read using the lctl
command.

For example:

    # Read number of unstable pages contained by each OSC
    lctl get_param osc.*.unstable_stats

The motivation for this change is in anticipation of implementing a
"soft sync" functionality, urging servers to commit these unstable
pages to stable storage. The per OSC accounting allows a client to
limit the soft sync request to only the OSCs which have outstanding
unstable pages.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: Ife1f90515cb60ba9f85daba5d50483562f2697d1
Patch-Set: 7
LU-2139 osc: Use SOFT_SYNC to urge server commit

This change adds a BRW page flag, OBD_BRW_SOFT_SYNC. This flag is
intended to urge a server to commit a client's unstable pages to
stable storage. A client will add this flag to any BRW requests while
it is in a state where it has "many" unstable pages pinned in its cache.

The server, upon receiving a page with this flag set, *should* begin
an async commit. The idea being that, with the proactive async commit,
the client's pinned unstable pages will transition into a stable state
faster than they would have otherwise. Although, the server side portion
of this agreement is still unimplemented, so the OBD_BRW_SOFT_SYNC flag
will currently fall on deaf ears.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I08946a5ec6afab390655643f6d3c8320bf010aea
Patch-Set: 12
LU-2139 ofd: Do async commit if SOFT_SYNC is seen

This change implements the server side portion of the OBD_BRW_SOFT_SYNC
agreement. When this flag is seen on an incoming BRW page, the server
will begin an asynchronous commit operation via dt_commit_async.

The intention is for this operation to commit pages belonging to a
client which has "too many" outstanding unstable pages in its cache.
Upon the commit operation completing, the last_commited value should
percolate out to the client of interest, allowing it to drop some of
its pinned unstable pages.

Unfortunately, there is currently no optimization to proactively send
the last_committed value to the client immediately after the commit
operation succeeds. It must be piggybacked on any future IO or
pings made by the client.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Change-Id: I1bac4bc517ab023a5b298f34bb695c7ba8ece896
Patch-Set: 2