ORI-565 osd: fixes for the local storage lib

- replace assertion with error handling
- do new fid allocation and disk update atomically

Signed-off-by: Mikhail Pershin <tappro@whamcloud.com>
Change-Id: I842d566f54fb735affe6eaa03913cda9affe8ba5
Reviewed-on: http://review.whamcloud.com/2280
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
ORI-491 llog: Fix object references of named logs

The llog module inserts named logs into loc_dir but does not add
corresponding references.  This causes link count problems that are
visible when running e2fsck against an ldiskfs-based MDT.  This patch
adds the missing reference when creating a named log.

Change-Id: I5a4ca34ed897f0ff8b42c263f1dfe38f2422b74b
Signed-off-by: Li Wei <liwei@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/2225
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
ORI-487 tests: print verbose fsx logs, run less

Because dual-mount fsx is taking so long to complete (average only
0.5 operation per second) it is desirable to figure out if there is
a particular operation that is causing the slowdown or not.  Enable
status updates for every operation on ZFS to monitor slow operations.

Also, in order to allow subtests after test_16 to be run, reduce
the number of operations to 500 for ZFS test runs until this bug
is fixed.

Conversely, for SLOW=yes runs on ldiskfs it is desirable to run fsx
longer, since this is a very valuable test for corner cases.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I339bca0f5857c17d958a781d1648ed34e040cf92
Reviewed-on: http://review.whamcloud.com/2253
Tested-by: Hudson
ORI-393 zfs-osd: basic zerocopy support

osd_get_bufs_read() grabs dbufs and list corresponding pages
in local niobuf array.

osd_get_bufs_write() loan an arc buffer for full block writes,
then assign them into arc (in commit path). in case of partial
block write osd_get_bufs_write() just allocate pages and use
them in bulks as before.

Change-Id: I4c9837491fe234bfc1751fc92e0f4d48f1178d98
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/2141
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Tested-by: Hudson
ORI-507 osd-zfs: Multiple OIs

This change implements multiple OIs for ZFS OSDs.  While it was
based on what was done for ldiskfs that actual implementation
differs.  However, at the core the implementation still relies
on a simple power of two lookup table to map fids->OIs.

This issue is related LU-822, see commit 4db03b9 which contains
the original ldiskfs implemenation.

Change-Id: I16164ae2805e391556fa38f4684a7802a25df105
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-on: http://review.whamcloud.com/2173
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
LU-1205 tests: add timestamps to sanityn 18 mmap

The sanityn.sh test_18 mmap_sanity.c test sometimes takes over
an hour to run, and sometimes only seconds.  Add timestamps to
the subtest results so that it is possible to debug where that
time is being spent.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ie33c46d9b72083c79112455ac9df6f357dc7be7a
Reviewed-on: http://review.whamcloud.com/2289
Tested-by: Hudson
ORI-166 config: return back to lov/osc in config profiles

for compatibility purposes we still use LOV/OSC in config.
profiles. mgc changes records on-fly to setup LOD/OSP on
MDS instead. also, MDT setups MDD manually.

mdd, lod, osp to recognize old configurations and
generate actual device names to support 1.8 configs.

Change-Id: I21046604e5e14aded3b109546c0cb3b219d18972
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: http://review.whamcloud.com/2101
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Hudson
ORI-564 osd-zfs: do not call into DMU holding rwlock

sa_update() and sa_object_size() can block within,
leading to deadlock if oo_attr_lock is held.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Idd8199c64da88c557f040ec564867996bf58780f
Reviewed-on: http://review.whamcloud.com/2332
Reviewed-by: Mike Pershin <tappro@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Tested-by: Hudson
ORI-507 osd-zfs: osd_oi_probe() to use right index

replace i with *count

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: Ia8250bcf821cd33f4f2cbf538c60858d469087af
Reviewed-on: http://review.whamcloud.com/2333
Reviewed-by: Mike Pershin <tappro@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Tested-by: Hudson
ORI-161 tests: Refresh conf-sanity 32[ab]

Existing conf-sanity 32[ab] does not run on multi-node clusters or
network types other than TCP.  This patch rewrites the tests to start
the targets on the MDS and mount the upgraded file system on the primary
client.  This scheme works both on a single-node development environment
and a typical Autotest cluster.

The disk image tarball format has been updated to include more
"metadata".  For example, the 1.8.7-wc1 and 2.1.1 tarballs added by this
patch include the kernel versions, architectures, Lustre versions, and
commit SHAs (currently only Lustre version) used to create the disk
images.  In addition, the script used to create the tarballs is recorded
as a test in test_32newtarball().

A couple of new data verifications are added to make sure the name
space, the file data, and most of the file attributes are consistent
across upgrades.

Change-Id: Iabdc3a0accb9ee4db24eae2755a66ad633f96109
Signed-off-by: Li Wei <liwei@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
ORI-541 osd-zfs: Release SA hold before dmu_tx_commit

This patch ensures that the SA handle does not have a hold on the
spill block at commit time.  If it does the extra reference will
cause the txg_sync_thread to treat the dbuf as unstable.  The
txg_sync_thread will then make a copy of the dbuf to ensure that
no subsequent updates sneak in to the syncing txg.

This also has the effect of leaving the original SA spill block
dbuf dirty.  If enough of these dirty dbufs accumulate on the
system it can cause dmu_tx_assign() to fail while it attempts
to sync out the dirty data.  If these dbufs are not attached to
transactions then they will not be written and the entire dmu
transaction processing system will effectively block forever.

To avoid this the best thing to do is to track all the objects
which are dirtied as part of the txg.  Then in osd_trans_stop()
walk the list of objects and drop SA handle reference on the
spill block.  Therefore the spill block dbuf won't appear
unstable and dirty buffers will not accumulate.

This change required that whenever possible the osd_thandle
structure be available for the dirty object to be linked too.
In the cases where that wasn't possible a private SA handle
was used and then immediately destroyed.

For places when this extra overhead was consider too much the
code was changed to use a cached value.  This was done once for
the object size in ost_read().  It was also done in several
places for the xattr zap object id.  In all these cases the
cached value is kept up to date and can be trusted.

Change-Id: Ia447aa1ea14704857a267095bf81d526ffc23964
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed-on: http://review.whamcloud.com/2259
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
ORI-484 obdclass: Remove EXPORT_SYMBOLs for static functions

Remove two EXPORT_SYMBOLs for static scope functions.  The ppc64
compiler catches and throws an error.

Signed-off-by: Christopher J. Morrone <morrone2@llnl.gov>
Change-Id: I0ead83bee397dd064e6a72e00a97c14cd950c8a0
Reviewed-by: Christopher J. Morrone <morrone2@llnl.gov>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
ORI-352 Remove cursory Solaris support

Several Solaris compatibility changes were added to Lustre when
it was being ported to Solaris.  However, since this work is
unlikely to be completed it's now just deadcode which should
be removed to improve readability.

Change-Id: Icbab8da71e4cbe21462657904e7e31afdeec184e
Signed-off-by: Brian Behlendorf <behlendorf1@llnl.gov>
ORI-560 osd-zfs: implement ZFS inode accounting

This patch adds tracking for inode usage in special ZAPs maintained
by Lustre itself. One index is created for user accounting and another
one for groups.
By default, lfs quota retrieves inode usage from the new accounting
ZAPs, but this can be changed via procfs (iused_estimate proc file
under OSD directory) to rely on the inode count estimate from the
block usage.

Signed-off-by: Johann Lombardi <johann@whamcloud.com>
Change-Id: I8cc487b11a52ef4db86af4a03e8ebfdf01917e5c
Reviewed-on: http://review.whamcloud.com/2385
LU-974 protocol: change OBD_CONNECT_GRANT_PARAM

Change the OBD_CONNECT_GRANT_PARAM flag value to avoid conflict
with the OBD_CONNECT_UMASK flag from LU-974.  While that patch is
not yet landed to our release tree, it is in use in production at
some customers.  While the risk of conflict is currently low, it
is easier to change the GRANT_PARAM value since only in use on the
orion branch, and isn't even handled by the client there yet.

Add (hopefully) clear comments for OBD_CONNECT and obd_connect_data
to ensure that they are not modified in some incompatible way across
branches.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I503892c3b595c0272b0941fa58a16a49631e80c6
Reviewed-on: http://review.whamcloud.com/2296
Reviewed-by: Johann Lombardi <johann@whamcloud.com>
Tested-by: Johann Lombardi <johann@whamcloud.com>
ORI-560 osd-zfs: create inode acct indexes in top-level ZAP

This patch moves the new ZAPs used for ZFS inode accounting outside
the ZPL namespace. The new index files are now attached to the
top-level ZAP (i.e. MASTER_NODE_OBJ) in order not to confuse ZPL which
might interpret those indexes as directories and assume the values are
object IDs.

Signed-off-by: Johann Lombardi <johann@whamcloud.com>
Change-Id: I16d58b0ce8459adcbb71f1803d3c5d34fb0d808f
Reviewed-on: http://review.whamcloud.com/2405
Tested-by: Hudson
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
ORI-611 osd-zfs: [UG]ID is stored in hexa in DMU acct objs

The [UG]ID is stored in hexadecimal format in the DMU accounting
objects, so we should pass base=16 to simple_strtoull() instead of
base=10. This only impacts accounting information dumped via procfs.

Signed-off-by: Johann Lombardi <johann@whamcloud.com>
Change-Id: I2652ff6e6be25d008ad21935eb76aef8aadfb41e
Reviewed-on: http://review.whamcloud.com/2413
Reviewed-by: Niu Yawei <niu@whamcloud.com>
Tested-by: Hudson
ORI-588 mdd: __mdd_acl_init() to set default ACL

the call to mdo_attr_set() could reset rc which actually
stores return code from lustre_posix_acl_create_masq() to
be used later.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I0b78cee8ccc09a6189ea09baabc163580be3c6fc
Reviewed-on: http://review.whamcloud.com/2416
Reviewed-by: Mike Pershin <tappro@whamcloud.com>
Tested-by: Hudson
ORI-610 osd-zfs: define attributes in the same order as ZPL

The accounting callback (zfs_space_delta_cb()) relies on static
offsets to look up the UID & GID in system attributes. It also does
not seem to support UID/GID relocated to the spill block.
To work around the problem, we define attributes in the same order as
SA_*_OFFSET until the ZFS accounting callback is fixed.

Signed-off-by: Johann Lombardi <johann@whamcloud.com>
Change-Id: I97c8d60ee1e2ec4e182fa4cbe3e6c629b4f3da5b
Reviewed-on: http://review.whamcloud.com/2428
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
ORI-568 osd: modify OI table to store pointers

Each OI is allocated separately and oi_table consists of pointers

Signed-off-by: Mikhail Pershin <tappro@whamcloud.com>
Change-Id: Ic9675eeac8fee103c199a7d63f3e7e61e0e73b0b
Reviewed-on: http://review.whamcloud.com/2421
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>