resize2fs: don't require fsck to print min size

My previous change ended up requiring that the filesystem
be fsck'd after the last mount, even if we are only querying
the minimum size.  This is a bit draconian, and it burned
the Fedora installer, which wants to calculate minimum size
for every filesystem in the box at install time, which in turn
requires a full fsck of every filesystem.

Try this one more time, and separate out the tests to make things
a bit more clear.  If we're only printing the min size, don't
require the fsck, as this is a bit less dangerous/critical.

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
badblocks: Limit maximum number of bad blocks

Currently maximum number of bad blocks is not limited in any way.
However our code can really handle at most INT_MAX/2 bad blocks (for
larger numbers binary search indexes start overflowing). So report
number of bad blocks is just too big instead of plain segfaulting.

It won't be too hard to raise the limit but I don't think there's any
real use for disks with over 1 billion of bad blocks...

Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Merge branch 'maint' into next
resize2fs: use old_fs to detect per-bg metadata blocks to free

When shrinking a filesystem, resize2fs wants to free per-bg metadata
blocks that are no longer needed.  This behavior is gated on whether
there's a superblock in the group as told by new_fs.  The check really
should be against old_fs, since we're effectively freeing blocks out
of old_fs in the transition to new_fs, but prior to sparse_super2 this
didn't matter since superblocks didn't move, so it didn't matter.

Under sparse_super2, however, there's a superblock in the last group,
so now we need to change the test to use old_fs as it should.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
resize2fs: don't interpret bitmap shift while crossing flexbg as raid stride

resize2fs tries to infer the RAID stride by observing differences
between the locations of adjacent block groups' block and inode
bitmaps within the block group.  If the two block groups being
compared belong to different flexbgs, however, it'll be fooled by the
large offset into thinking that the FS has an abnormally large RAID
stride.

Therefore, teach it not to get confused by crossing a flexbg.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reported-by: TR Reardon <thomas_reardon@hotmail.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
resize2fs: set bg flags and unused inode count when resizing

Recalculate the unused inode count and the block/inode uninit flags
when resizing a filesystem.  This can speed up future e2fsck runs
considerably and will reduce mount times.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
resize2fs: don't play stupid games with the block count

While it may be true that playing games with old_fs' block count
during a grow operation shuts up a bunch of warnings, resize2fs
doesn't actually expand the group descriptor array to match the size
we're artificially stuffing into old_fs, which means that if we
actually need to allocate a block out of the larger fs (i.e. we're in
desperation mode), ext2fs_block_alloc_stats2() scribbles on the heap,
leading to crashes if you're lucky and FS corruption if not.

So, rip that piece out and turn off com_err warnings properly and add
a test case to deal with growing a nearly full filesystem.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Bump version.h for an experimental release

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
libext2fs: speed up the max extent depth api call

The maximum extent tree depth really only depends on the filesystem
block size, so cache the last result if possible.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
resize2fs: convert fs to and from 64bit mode

resize2fs does its magic by loading a filesystem, duplicating the
in-memory image of that fs, moving relevant blocks out of the way of
whatever new metadata get created, and finally writing everything back
out to disk.  Enabling 64bit mode enlarges the group descriptors,
which makes resize2fs a reasonable vehicle for taking care of the rest
of the bookkeeping requirements, so add to resize2fs the ability to
convert a filesystem to 64bit mode and back.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Cc: TR Reardon <thomas_reardon@hotmail.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
tests: test resize2fs 32->64 and 64->32bit conversion code

Add some simple tests to check that flex_bg and meta_bg filesystems
can be converted between 32 and 64bit layouts.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
misc: fix clang warnings

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
libext2fs: zero blocks via FALLOC_FL_ZERO_RANGE in ext2fs_zero_blocks

Plumb a new call into the IO manager to support translating
ext2fs_zero_blocks calls into the equivalent FALLOC_FL_ZERO_RANGE
fallocate flag primitive when possible.  This patch provides _only_
support for file-based images.

v2: Remove zero-out for block devices until BLKZEROOUT is cleaned up.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
libext2fs: ext2fs_new_block2() should call alloc_block hook

If ext2fs_new_block2() is called without a specific block map, we
should call the alloc_block hook before checking fs->block_map.  This
helps us to avoid a bug in e2fsck where we need to allocate a block
but instead of consulting block_found_map, we use the FS bitmaps,
which (prior to pass 5) could be wrong.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
tune2fs: disable csum verification before resizing inode

When we're turning on metadata checksumming /and/ resizing the inode
at the same time, disable checksum verification during the
resize_inode() call because the subroutines it calls will try to
verify the checksums (which have not yet been set), causing the
operation to fail unnecessarily.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
resize2fs: don't mark unallocated bg metadata blocks when fixing bg flags

When fixing up block group bitmaps on the new fs, don't try to mark
unallocated group metadata blocks; these will be allocated (and
marked) later.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
libext2fs/e2fsck: provide routines to read-ahead metadata

This patch adds to e2fsck the ability to pre-fetch metadata into the
page cache in the hopes of speeding up fsck runs.  There are two new
functions -- the first allows a caller to readahead a list of blocks,
and the second is a helper function that uses that first mechanism to
load group data (bitmaps, inode tables).

These new e2fsck routines require the addition of a dblist API to
allow us to iterate a subset of a dblist.  This will enable
incremental directory block readahead in e2fsck pass 2.

There's also a function to estimate the readahead given a FS.

v2: Add an API to create a dblist with a given number of list elements
pre-allocated.  This enables us to save ~2ms per call to
e2fsck_readahead() (assuming a 2MB RA buffer) by not having to
repeatedly call ext2_resize_mem as we add blocks to the list.

v3: Instead of creating dblists of arbitrary size, change the dblist
iterator to allow iterating a sub-range.  This eliminates a lot of
unnecessary list copying during e2fsck part2.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
e2fsck: read-ahead metadata during passes 1, 2, and 4

e2fsck pass1 is modified to use the block group data prefetch function
to try to fetch the inode tables into the pagecache before it is
needed.  We iterate through the blockgroups until we have enough inode
tables that need reading such that we can issue readahead; then we sit
and wait until the last inode table block read of the last group to
start fetching the next bunch.

pass2 is modified to use the dirblock prefetching function to prefetch
the list of directory blocks that are assembled in pass1.  We use the
"iterate a subset of a dblist" and avoid copying the dblist.  Directory
blocks are fetched incrementally as we walk through the directory
block list.  In previous iterations of this patch we would free the
directory blocks after processing, but the performance hit to e2fsck
itself wasn't worth it.  Furthermore, it is anticipated that most
users will then mount the FS and start using the directories, so they
may as well remain in the page cache.

pass4 is modified to prefetch the block and inode bitmaps in
anticipation of pass 5, because pass4 is entirely CPU bound.

In general, these mechanisms can decrease fsck time by 10-40%, if the
host system has sufficient memory and the storage system can provide a
lot of IOPs.  Pretty much any storage system capable of handling
multiple IOs in-flight at any time will see a fairly large performance
boost.  (Single-issue USB mass storage disks seem to suffer badly.)

By default, the readahead buffer size will be set to the size of a block
group's inode table (which is 2MiB for a regular ext4 FS).  The -E
readahead_kb= option can be given to specify the amount of memory to
use for readahead or zero to disable it entirely; or an option can be
given in e2fsck.conf.

v2: Fix an off-by-one error in the pass1 readahead which made the
readahead trigger one inode too late if the block groups are full.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
e2fsck: rebuild sparse extent trees/convert non-extent ext3 files

Teach e2fsck to construct extent trees.  This enables us to do either
of the following: compress a highly sparse extent tree into fewer ETB
blocks; or convert a ext3-style block mapped file to an extent file.

For files that are already extent based, this algorithm will only run
if pass1 determines either (1) that a whole level of extent tree will
fit into a higher level of the tree; (2) that the size of any level
can be reduced by at least one ETB block; or (3) the extent tree is
unnecessarily deep.  It will not run at all if errors are found and
the user declines to fix the errors.

For block-mapped files, conversion only happens if the extent feature
is enabled and "-E bmap2extent" is passed to e2fsck.  It will not run
at all if errors are left unfixed.  After conversion, files larger
than 12 blocks should be defragmented to eliminate empty holes where a
block lives.

The extent tree constructor is pretty dumb -- it creates a list of
leaf extents (adjacent extents are collapsed), marks all indirect
blocks / ETB blocks free, installs a new extent tree root in the
inode, then loads the leaf extents into the tree.

v2: Account for extent tree block slack that we create when splitting
a block, so that we don't repeatedly annoy the user to rebuild a tree
that we can't optimize further.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
tests: verify proper rebuilding of sparse extent trees and block map file conversion

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>