btrfs-progs: add quota group verify code

This patch adds functionality (in qgroup-verify.c) to compute bytecounts in
subvolume quota groups. The original groups are read in and stored in memory
so that after we compute our own bytecounts, we can compare them with those
on disk. A print function is provided to do this comparison and show the
results on the console.

A 'qgroup check' pass is added to btrfsck. If any subvolume quota groups
differ from what we compute, the differences for them are printed.  We also
provide an option '--qgroup-report' which will run only the quota check code
and print a report on all quota groups.  Other than making it possible to
verify that our qgroup changes work correctly, this mode can also be used in
xfstests for automated checking after qgroup tests.

This patch does not address the following:
- compressed counts are identical to non compressed, because kernel doesn't
  make the distinction yet.  Adding the code to verify compressed counts
  shouldn't be hard at all though once kernel can do this.
- It is only concerned with subvolume quota groups (like most of
  btrfs-progs).

Signed-off-by: Mark Fasheh <mfasheh@suse.de>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: fix uninitialized number count in chunk-recover

When count the number of unordered device extents in chunk-recover,
the counter should be reinitialized to be used.
Also, introduce a new function for the counting job.

Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: fix blindly goto failure for chunk-recover

If the csum of one stripe is not able to judge the order of two
device extents, the stripe may happen to belong to the device extent
that is already kicked out as ordered.
Take this condition into consideration, don't report failure and
give more tries with the stripes following.

Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Btrfs-progs: fsck: fix wrong check for btrfs_read_fs_root()

When encountering a corrupted fs root node, fsck hit following message:

Check tree block failed, want=29360128, have=0
Check tree block failed, want=29360128, have=0
Check tree block failed, want=29360128, have=0
Check tree block failed, want=29360128, have=0
Check tree block failed, want=29360128, have=0
read block failed check_tree_block
Checking filesystem on /dev/sda9
UUID: 0d295d80-bae2-45f2-a106-120dbfd0e173
checking extents
Segmentation fault (core dumped)

This is because in btrfs_setup_all_roots(), we check
btrfs_read_fs_root() return value by verifing whether it is
NULL pointer, this is wrong since btrfs_read_fs_root() return
PTR_ERR(ret), fix it.

Signed-off-by: Wang Shilong <wangsl.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
Btrfs-progs: fsck: add an option to check data csums

This patch adds an option '--check-data-csum' to verify data checksums.
fsck won't check data csums unless users specify this option explictly.

Signed-off-by: Wang Shilong <wangsl.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: update man page for btrfs-show-super

Add '-f' option for btrfs-show-super manpage,
This option implies that sys chunk array and backup roots info
will show up.

Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: Documentation: Add filter section for btrfs-balance.

Man page for 'btrfs-balance' mentioned <filters> but does not explain
them, which make end users hard to use '-d', '-m' or '-s options.

This patch will use the explanations from
https://btrfs.wiki.kernel.org/index.php/Balance_Filters to enrich the
man page.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: make pretty Documentation/ build match the rest

This is the most important patch ever.  ;)

I found this to be less aesthetically pleasing than it was before:

     [CC]     btrfstune.o
 Making all in Documentation
     ASCIIDOC btrfs-convert.xml
     [LD]     btrfstune
     XMLTO btrfs-convert.8
     [CC]     btrfs-show-super.o
     GZIP btrfs-convert.8.gz
     [LD]     btrfs-show-super
     ASCIIDOC btrfs-debug-tree.xml
     XMLTO btrfs-debug-tree.8

so I shortened the pretty-text to match what we had before.

Also, make clean "quiet" like it is in the top dir.

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
[Changed to ASCII and XMLTO]
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: canonicalize pathnames for device commands

mount(8) will canonicalize pathnames before passing them to the kernel.
Links to e.g. /dev/sda will be resolved to /dev/sda. Links to /dev/dm-#
will be resolved using the name of the device mapper table to
/dev/mapper/<name>.

Btrfs will use whatever name the user passes to it, regardless of whether
it is canonical or not. That means that if a 'btrfs device ready' is
issued on any device node pointing to the original device, it will adopt
the new name instead of the name that was used during mount.

Mounting using /dev/sdb2 will result in df:
/dev/sdb2      209715200 39328 207577088   1% /mnt

lrwxrwxrwx 1 root root 4 Jun  4 13:36 /dev/whatever-i-like -> sdb2
/dev/whatever-i-like 209715200 39328 207577088   1% /mnt

Likewise, mounting with /dev/mapper/whatever and using /dev/dm-0 with a
btrfs device command results in df showing /dev/dm-0. This can happen with
multipath devices with friendly names enabled and doing something like
'partprobe' which (at least with our version) ends up issuing a 'change'
uevent on the sysfs node. That *always* uses the dm-# name, and we get
confused users.

This patch does the same canonicalization of the paths that mount does
so that we don't end up having inconsistent names reported by ->show_devices
later.

Signed-off-by: Jeff Mahoney <jeffm@suse.com>
[use PATH_MAX in canonicalize_dm_name]
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: cleanup unused assignment for chunk-recover

The 'num_unordered' will be recounted after 'goto out',
just remove it.

Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: add mount options to btrfs-mount.5

This is a straight cut and paste from the util-linux
mount manpage into btrfs-mount.5

It's pretty much impossible for util-linux to keep up
with every filesystem out there, and Karel has more than
once expressed a wish that mount options move into fs-specific
manpages.

So, here we go.

The way btrfs asciidoc is generated, there's not a trivial
way to have both btrfs(5) and btrfs(8) so I named it btrfs-mount(5)
for now.  A bit ick and I'm open to suggestions.

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: rename btrfs-mount.5 to btrfs.5

We'd like to keep the name of category 5 page same as the filesystem
to be consistent with the rest.

Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: Fix undefined behavior in radix-tree.c.

When running with UndefinedBehaviorSanitizer, the tests produce the following
error:

  radix-tree.c:836:30: runtime error: shift exponent 18446744073709551613
  is too large for 64-bit type 'unsigned long'

(That's a negative shift exponent represented as an unsigned long.)

Even though the value is discarded in those cases, it's still undefined
behavior; see the C99 standard, section 6.5.7, paragraph three: "If the
value of the right operand is negative [...] the behavior is undefined."

Signed-off-by: Adam Buchbinder <abuchbinder@google.com>
Reviewed-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: Fix a use-after-free in the volumes code.

When a struct btrfs_fs_devices was being torn down by
btrfs_close_devices(), there was an invalidated pointer in the global
list fs_uuids which still pointed to it; if a device was closed and
then reopened (which btrfs-convert does), freed memory would be
accessed.

This was found using ThreadSanitizer (pretty much doing what
AddressSanitizer would, but not exiting after the first failure).
To reproduce, build with -fsanitize=thread and run 'make test'.
Representative output is below.

This change makes the current tests TSan-clean.

WARNING: ThreadSanitizer: heap-use-after-free (pid=29161)
  Read of size 8 at 0x7d180000eee0 by main thread:
    #0 memcmp ??:0
    #1 find_fsid .../volumes.c:81
    #2 device_list_add .../volumes.c:95
    #3 btrfs_scan_one_device .../volumes.c:259
    #4 btrfs_scan_fs_devices .../disk-io.c:1002
    #5 __open_ctree_fd .../disk-io.c:1090
    #6 open_ctree_fd .../disk-io.c:1191
    #7 do_convert .../btrfs-convert.c:2317
    #8 main .../btrfs-convert.c:2745

  Previous write of size 8 at 0x7d180000eee0 by main thread:
    #0 free ??:0
    #1 btrfs_close_devices .../volumes.c:191
    #2 close_ctree .../disk-io.c:1401
    #3 do_convert .../btrfs-convert.c:2300
    #4 main .../btrfs-convert.c:2745

  Location is heap block of size 96 at 0x7d180000eee0 allocated by main thread:
    #0 calloc ??:0 (exe+0x00000002acc6)
    #1 device_list_add .../volumes.c:97
    #2 btrfs_scan_one_device .../volumes.c:259
    #3 btrfs_scan_fs_devices .../disk-io.c:1002
    #4 __open_ctree_fd .../disk-io.c:1090
    #5 open_ctree_fd .../disk-io.c:1191
    #6 do_convert .../btrfs-convert.c:2256
    #7 main .../btrfs-convert.c:2745

Signed-off-by: Adam Buchbinder <abuchbinder@google.com>
Reviewed-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: update manpage for btrfs resize support size unit t/p/e

btrfs resize now support size unit parse of k/m/g/t/p/e in kernel space,
adopt the changes in userspace manpage.

Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: Add human readable flags output string for extent flags.

Current btrfs-debug-tree outputs extent flags as numbers,
which makes it hard to understand and need to check the source to
understand the meaning.

This patch will convert numberic flags output to human readable strings.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: Add human readable flags output for chunk/block group type.

Current btrfs-debug-tree output chunk/block group type as numbers,
which makes it hard to understand and need to check the source to
understand the meaning.

This patch will convert numeric type output to human readable strings.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: Add some simple end-to-end tests for btrfs-convert

These use the system's mke2fs, and don't require loop devices
or root privileges.

They don't pick up anything with the default flags right now,
but they do pick up some sanitizer issues when the tools are
compiled with any of -fsanitize={address,memory,thread}.

Signed-off-by: Adam Buchbinder <abuchbinder@google.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: fix missing parity stripe for raid6 in chunk-recover

When deal with the p & q stripes for data profile raid6, chunk-recover
forgets to insert them into the chunk record. Just insert them back
freely.
Also wrap the insert procedure into a new function, fill_chunk_up.

Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>
btrfs-progs: fix max mirror number error for chunk-recover

When run chunk-recover on a health btrfs(data profile raid0, with
plenty of data), the program has a chance to abort on the number
of mirrors of an extent.

According to the kernel code, the max mirror number of an extent
is 3 not 2:
	ctree.h: 		BTRFS_MAX_MIRRORS	3
	chunk-recover.c :	BTRFS_NUM_MIRRORS	2
just change BTRFS_NUM_MIRRORS to 3, and everything goes well.

Signed-off-by: Gui Hecheng <guihc.fnst@cn.fujitsu.com>
Signed-off-by: David Sterba <dsterba@suse.cz>