LU-1943 ofd: save server last transaction number

After reading server data from the last_rcvd file we must store the
last transaction number in lu_target.  Otherwise we write a zero last
transaction number back to last_rcvd, causing transaction numbers to
roll back to 2^32 every time the server is restarted.  This can lead
to 'server went back in time' client console errors.

Whamcloud-bug-id: ORI-670
Signed-off-by: Ned Bass <bass6@llnl.gov>
Signed-off-by: Mikhail Pershin <tappro@whamcloud.com>
Change-Id: Iaff4af3e4bd6fce0da1d1b0886147538c3300c1e
Reviewed-on: http://review.whamcloud.com/4190
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Ian Colle <Ian.Colle@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1842 quota: add quota locks support on QMT

This patch adds methods for the quota lock lvb management and
implements an intent policy for quota locks.
It also introduces the rebalance thread which is in charge of
sending glimpse callbacks to slave to notify them of new quota
settings.

This patch also enables sanity-quota.sh test 36 which should now
pass.

Signed-off-by: Johann Lombardi <johann.lombardi@intel.com>
Change-Id: I0cf7f86d480296f7e8fe774ed8937aae27736f88
Reviewed-on: http://review.whamcloud.com/4166
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-1842 quota: add acq/rel logic on QMT

This patch adds the quota acquire/release logic on the QMT.
The QMT is in charge of distributing quota space across quota
slaves. It grants a significant amount of quota space (75%) initially
and then tries to balance spare quota space among slaves.
To do so, it sends glimpse on per-ID lock to notify slaves of qunit
shrink. qunit is now the maximal amount of available quota space a
slave can own. As for qtune, this value (typically 1/4 of qunit) is
used on the quota slave to decide when to release/pre-acquire quota
space.

Signed-off-by: Johann Lombardi <johann.lombardi@intel.com>
Change-Id: I8064ce614955421681ca1da7e2bb774001ca3bea
Reviewed-on: http://review.whamcloud.com/4178
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-2033 scrub: lookup server local files by name

after backup/restore ldiskfs's OI becomes invalid (inodes
change). OI scrubber scans a filesystem to rebuild OI. this
can take quite long. to be able to start MDS earlier (right
after restore) MDS should lookup by name all internal files
required for startup as name->inode mapping is preserved by
backup/restore. this patch make local object library to
lookup a file storing last used fid within a given sequence.

the patch also fixes sanity-scrub/11 to expect skipping of
files created + directory + per-ost llog objects.

Signed-off-by: Fan Yong <yong.fan@whamcloud.com>
Change-Id: I56179c03279062464960af73de031f13d31bac14
Reviewed-on: http://review.whamcloud.com/4106
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1842 quota: add core QSD code

Add acquire/release logic on the qsd side as well as reintegration
which is handled in 3 steps:
- enqueue global index lock
- fetch global index and update local enforce list
- fetch slave index and update local copy
- adjust/release quota space as appropriate

This patch also fixes the ofd code to call ->ldo_recovery_complete
across the stack.

Signed-off-by: Johann Lombardi <johann.lombardi@intel.com>
Change-Id: Ib4a8910fe34de878159827c76b40a8f899aaad10
Signed-off-by: Niu Yawei <yawei.niu@intel.com>
Reviewed-on: http://review.whamcloud.com/4182
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <niu@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1842 test: re-enable all sanity-quota tests

This patch enables all sanity-quota tests that were disabled in order
to land the new quota architecture (based on qsd/qmt). All those
tests should now pass.

This patch also reduces the time it takes to run test 6 (it used to
sleep for at_max = 600s).

Signed-off-by: Johann Lombardi <johann.lombardi@intel.com>
Change-Id: Idac4812663a3c90a213cc59d03b5f28f561ac39d
Reviewed-on: http://review.whamcloud.com/4200
Tested-by: Hudson
Reviewed-by: Jinshan Xiong <jinshan.xiong@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-1842 quota: don't setup qsd for standalone MGS

Only set up the qsd instance if the associated target is either a MDT
or a OST. We typically have no interest in setting up a QSD instance
on a standalone MGS.

Signed-off-by: Johann Lombardi <johann.lombardi@intel.com>
Change-Id: I0facb08feef0b36f141b749d7be538cbde5f1114
Reviewed-on: http://review.whamcloud.com/4203
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Niu Yawei <niu@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1943 tests: unlink all files when mdsrate times out

Port of ORI-453 to the master

When mdsrate cannot create the requested number of files in the
allocated timeframe (i.e. the --time option), filenames are not
necessarily consecutive and can have an index higher than the
number of files that was actually created. Here is an example:

0: iters 0 nfiles 100000 time 1 begin 0 end 99998 dirthreads 2.
1: iters 0 nfiles 100000 time 1 begin 1 end 99999 dirthreads 2.
0: client-4.lab.whamcloud.com starting at Mon Jan  9 08:26:07 2012
1: client-12.lab.whamcloud.com starting at Mon Jan  9 08:26:07 2012
0: created 1884 files, last file 'f3766'.
Rate: 2470.42 eff 2470.59 aggr 1235.29 avg client creates/sec
     (total: 2 threads 2440 creates 1 dirs 2 threads/dir 0.99 secs)
0: client-4.lab.whamcloud.com finished at Mon Jan  9 08:26:08 2012
1: created 556 files, last file 'f1111'.
1: client-12.lab.whamcloud.com finished at Mon Jan  9 08:26:08 2012

2440 files have been created and the last file name was f1111 for
rank 1 and f3766 for rank 0. Running mdsrate --unlink --nfiles 2440
in this case would discard all files with an index >= 2440.

With this patch, each mdsrate task will complete properly and report
the right number of executed operation as soon as one file is missing
in the per-task sequence. This behavior can be changed with the
--ignore and --recreate option which are already available.
This way, mdsrate --unlink --nfiles 100000 will work fine in the
example above since it will unlink all the existing files, without
doing tons of useless lookups.

Signed-off-by: Johann Lombardi <johann@whamcloud.com>
Signed-off-by: Mikhail Pershin <tappro@whamcloud.com>
Change-Id: I96f690ad0f36c65353bbc5a401455479b4c3783b
Reviewed-on: http://review.whamcloud.com/4195
Reviewed-by: Johann Lombardi <johann.lombardi@intel.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
LU-1943 tests: Allocate sequences on MOUNT2 in replay_barrier()

Port of ORI-448 to the master

In order to avoid sequence file updates after a target is made
read-only, replay_barrier() creates a file in "MOUNT" on every
client node. This is not enough because the "MOUNT2" clients may
also trigger sequence file updates. This patch makes sure the trick
is applied to "MOUNT2" clients as well.

Change-Id: I832689a2d2ca205d927bc5d0a15ab14fceb3bf80
Signed-off-by: Li Wei <liwei@whamcloud.com>
Reviewed-by: Mikhail Pershin <tappro@whamcloud.com>
Signed-off-by: Mikhail Pershin <tappro@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/4196
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
LU-1943 test: Support --with-ldiskfsprogs in test framework

ORI-347 port to the master

When Lustre is built with the --with-ldiskfsprogs option the
Lustre utilities will expect to find the e2fsprogs utilties
repackaged as ldiskfsprogs on the system.  This is done to
avoid having to replace the distribution provided e2fsprogs
package.

This change extends the test framework to check for the
ldiskfsprogs utilties to avoid having to explicitly set
the shell variables when --with-ldiskfsprogs is used.
If the tools are not available it will fallback to using
the e2fsprogs utilities.

Signed-off-by: Prakash Surya <surya1@llnl.gov>
Signed-off-by: Mikhail Pershin <tappro@whamcloud.com>
Change-Id: Ia094d9b005f7e455beca1c5d6d6ad03bdc03671f
Reviewed-on: http://review.whamcloud.com/4194
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-1607 tests: allow MODOPTS_LIBCFS module params

The changes made for forcing non-default CPT partition counts in
commit 95d67b28076f7938a6c962a5256e9b581a439f71 now prevents
overriding libcfs module options using MODOPTS_LIBCFS, because
there is always the cpu_npartitions parameter specified at module
loading time that overrides the environment variable.

Instead add the cpu_npartitions=2 argument to the environment
variable instead of making it a parameter.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I94bf500164dc23e81080bd2675646b5fa7500c1e
Reviewed-on: http://review.whamcloud.com/3999
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
LU-2093 osd: maintain O/<group>/LAST_ID properly

unfortunately this is not easy to do from ofd,
we have to do this in osd.

Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Change-Id: I389f278ac53da2a65ba8ed4a6e93cb9622eedae2
Reviewed-on: http://review.whamcloud.com/4210
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Oleg Drokin <green@whamcloud.com>