LU-3420 scrub: trigger OI scrub properly

In osd_fid_lookup(), if the iget() cannot locate the backend inode
(return -ENOENT or -ESTALE) with the found ino#/gen in the OI file,
there may be three cases:

1) MDT file-level backup/restore caused the OI invalid.
2) Someone unlinked the object but NOT removed the OI mapping, such
   as mount target device as ldiskfs, and modify something directly.
3) Someone just removed the object between the osd_oi_lookup() and
   the iget(). It is normal.

Under such case we can lookup OI again to check whether related OI
mapping is still there or not. If not, it is 3rd case. Otherwise,
it is diffcult to distinguish the 2nd from the 1st case. Relatively
speaking, the 1st case is more common than the 2nd case, so trigger
OI scrub without further distinguish.

To reduce the 2nd case, the initial OI scrub will scan the local
objects set to remove those stale OI mappings, such as the entry
for "CATALOGS" after file-level backup/restore.

Another update is about mapping FID to inode for the object with
remote name entry. For example, dir1's name entry resides on the
MDT0. Its object resides on the MDT1. On the MDT1, do file-level
backup/resotre. And before the OI scrub finishing to rebuilt the
OI files on the MDT1, MDT0 send RPC to MDT1 to access dir1's obj
with dir1's FID. On the MDT1, the OI mapping for FID1 is invalid
yet. Under such case, we have two possible solutions:

a) The MDT1 returns -EINPROGRESS to the MDT0, and the MDT0 retry
   such RPC sometime later.

b) On the MDT1, There is the directory "/REMOTE_PARENT_DIR" hold
   all the objects which have remote name entries on other MDTs.
   The object name under the "/REMOTE_PARENT_DIR" is FID string.
   So when get invalid OI mapping, OSD can localte the inode via
   lookup the "/REMOTE_PARENT_DIR" with the given FID.

Compared the two solutions, a) will cause the MDT0 busy wait until
related OI mapping rebuilt. b) is relative better. And bacause the
MDT file-level backup/restore is rare case, it will not affect the
normal cases for FID to inode mapping.

In fact, from the OSD view, it does not know whether the object for
the given FID is referenced by some remote name entry or not, and
especially for DNE II, a multiple-linked object may have many name
entries reside on many MDTs.

To simplify the operation, OSD will not distinguish more, it just
lookup the "/REMOTE_PARENT_DIR". Usually, it only happened for the
RPC from other MDT during the OI scrub, or for the client side RPC
with FID only, such as FID to path, or from old connected client.

Test-Parameters: testlist=sanity-scrub
Signed-off-by: Fan Yong <fan.yong@intel.com>
Change-Id: I503bd1e2d5d5eb3976a46c385ae041d11d3b9c32
Reviewed-on: http://review.whamcloud.com/6515
Tested-by: Hudson
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: wangdi <di.wang@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
LU-2662 build: pthread build issue

Adding strlcpy to lustre/utils/liblustreapi.c caused
unexpected build issues.

The build would fail with undefined refrence to
'pthread_create` in user-prim.c.

Adding the pthread flags to the makefile allow the
build to past. I am not sure this is the "BEST" way.

Signed-off-by: Keith Mannthey <keith.mannthey@intel.com>
Change-Id: I62ac4d5c53a42411ef2e8ff37a455308cffed167
Reviewed-on: http://review.whamcloud.com/5180
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Minh Diep <minh.diep@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-1346 libcfs: replace cfs_ memory wrappers

Replace memory relevant wrappers with kernel API.

Affected primitives:
CFS_PAGE_SIZE, CFS_PAGE_SHIFT, cfs_num_physpages,
cfs_copy_from_user, cfs_copy_to_user, cfs_page_address,
cfs_kmap/cfs_kunmap, cfs_get_page, cfs_page_count,
cfs_page_index, cfs_page_pin, cfs_page_unpin,
cfs_memory_pressure_get/set/clr, CFS_NUM_CACHEPAGES,
CFS_ALLOC_XXX flags, cfs_alloc/free, cfs_alloc/free_large,
cfs_alloc/free_page, CFS_DECL_MMSPACE, CFS_MMSPACE_OPEN,
CFS_MMSPACE_CLOSE, CFS_SLAB_XXX flags, cfs_shrinker_t,
cfs_set/remove_shrinker, CFS_DEFAULT_SEEKS, cfs_mem_cache_t,
cfs_mem_cache_alloc/free/create/destroy, cfs_mem_is_in_cache

manual changes:
1. cfs_alloc_flags_to_gfp() is removed
2. remove kmalloc/kfree etc. from linux-mem.c and linux-mem.h
3. remove page_address/kmap/kunmap etc. from linux-mem.h
4. remove page_cache_get/page_cache_release from echo_internal.h. They
are defined already in user-mem.h
5. change kmem_cache_create/destroy prototype to kernel's and modify
all callers to match them
6. define _SPL_KMEM_H and related macros to avoid using spl's
sys/kmem.h that redefines slab allocator
7. change kmem_virt to is_vmalloc_addr as provided by kernel, so that
we don't use any spl's sys/kmem.h functions
8. clean up include files a little bit in osd-zfs
9. various coding style cleanup

NUMA allocators(cfs_cpt_xxx) are not changed in this patch.
gnilnd is not converted, as requested by James Simmons.

Signed-off-by: Liu Xuezhao <xuezhao.liu@emc.com>
Signed-off-by: Peng Tao <tao.peng@emc.com>
Change-Id: Iadfbb0d5a0e31c78dd6c811e5ffdb468fa7e6f44
Reviewed-on: http://review.whamcloud.com/2831
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3365 lmv: support DNE with HSM.

Send HSM requests to the appropriate MDT. Split lists of fids of HSM
actions into one list per MDT.
Move kuc registration/unregistration from MDC to LMV as this is not
MDT related.

Signed-off-by: Thomas Leibovici <thomas.leibovici@cea.fr>
Change-Id: Iecba75dccbf15abc667dcca4800113c70a824d5c
Reviewed-on: http://review.whamcloud.com/6714
Tested-by: Hudson
Reviewed-by: John L. Hammond <john.hammond@intel.com>
Reviewed-by: jacques-Charles Lafoucriere <jacques-charles.lafoucriere@cea.fr>
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-3487 llite: handle io init failure in ll_fault_io_init()

In ll_fault_io_init(), if cl_io_init() has failed then cleanup and
return an ERR_PTR(). This fixes an oops in the page fault handling
code when a partially initialized io is used. In ll_page_mkwrite0() do
not call cl_io_fini() on an ERR_PTR().

Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: I04b1c1f3071413ba3fa53bbbfcb83fe325ed9050
Reviewed-on: http://review.whamcloud.com/6735
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>
LU-2903 tests: Increase zfs log size allowed in fs_log_size()

The log size in ZFS has been seen to be as much as 384.  This should
allow the affected tests pass without false-negatives.

Signed-off-by: Nathaniel Clark <nathaniel.l.clark@intel.com>
Change-Id: I9a1f4c25b9c14de0c3ea13f5064092fec86f7b46
Reviewed-on: http://review.whamcloud.com/6692
Tested-by: Hudson
Tested-by: Maloo <whamcloud.maloo@gmail.com>
Reviewed-by: Keith Mannthey <keith.mannthey@intel.com>
Reviewed-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>