Identify damaged files

Related articles
badblocks
File recovery
This article gives details on how to find out which file owns a given
disk sector
. The main purpose for doing so is finding out which file was damaged in the event a storage device develops any
bad sectors
(that way you will know if you lost anything important).
For most of these commands you will have to be either root or a user that has direct read access to the drive you are checking (being a member of the disk group should be enough). As usual, a current backup is always a good idea, especially if imminent drive failure is suspected.
S.M.A.R.T.
can help determining that.
Filesystems
btrfs
Unlike other filesystem types,
btrfs
has native support for reporting on damaged files. When scrubbing a partition,
btrfs
reads all data and metadata blocks and verifies checksums. It automatically repairs corrupted blocks if there is a correct copy available in a RAID configuration.
btrfs
also reports any unreadable sector along with the affected file via system log.
Find damaged files
Tip
Btrfs desktop notifications
help you quickly detect and respond to any file damage.
First of all,
scrub
the damaged partition:
# btrfs scrub start -Bd /dev/
sdxy
The same partition can be scrubbed when mounted by providing its mount point:
# btrfs scrub start -Bd
/mnt/point
Scrub reports can be retrieved as many times as need with the
status
subcommand:
# btrfs scrub status /mnt/point
scrub status for e11013b3-b244-4d1a-a9c7-3956db1a699c
        scrub started at Thu Apr 23 19:07:45 2015 and finished after 372 seconds
        total bytes scrubbed: 301.13GiB with 1 errors
        error details: read=1
        corrected errors: 0, uncorrectable errors: 1, unverified errors: 0
When done, search the kernel log for I/O errors:
The factual accuracy of this article or section is disputed.
Reason:
Is the sector still included? (Discuss in
Talk:Identify damaged files#Btrfs section may need updating
)
# journalctl --output=cat --grep='BTRFS .* i/o error' | sort | uniq | less
Jan 16 23:14:19 my_server kernel: BTRFS: i/o error at logical 398602014720 on dev /dev/sdxy, sector
4621320
, root 5, inode 23839, offset 4378624, length 4096, links 1 (path:
my/damaged/file
)
The output above reveals that sector
4621320
could not be read and it contains data for file
my/damaged/file
. The sector number can be directly used with
hdparm
when reallocating (see
#Force the disk to reallocate bad block
).
Ext2, ext3, and ext4
Debug the filesystem
The
tune2fs
command will give you access to all the low level structures within any ext2/
ext3
/
ext4
filesystem.
The first thing we want to do is get the block size from the filesystem in question. Just run:
# tune2fs -l /dev/sdxy | grep Block
Block count:              29119820
Block size:               4096
In this case 4096 is the block size being used (it appears to be the default).
If you did not run
badblocks
using the block size that your filesystem is using then you will need to convert your block number(s) to match it (remember to use the block number(s) relative to the partition they are on).
i.e. block number 100 with a block size of 1024 bytes becomes block number 25 at 4096 bytes. The formula is:
(original block number) / ((filesystem block size) / (badblocks block size))
Now the entire point of running this program (for the purpose of this article) is to get the
inode
number. Before continuing to debug it, unmount the partition. Then, run the command:
# debugfs
Then in the
debugfs
console, use the
open
command on the
ext
partition containing the bad sector:
debugfs:  open /dev/sdxy
Finally, use the
testb
command to get information about the block in question (in this example block 1000):
debugfs:  testb
blocknumber
Note
If
debugfs
says that block is not in use then that means the block is not allocated and is being used as free space. In that case this is a good thing as it means nothing important was damaged.
If the block is in use then run this command to get the
inode
number
icheck
blocknumber
This will return two numbers. The block number and the
inode
number.
Find damaged files
Use the
inode
number (second number from the
icheck
command) with the
ncheck
command:
ncheck
inodenumber
The
debugfs
console will give you the full pathname to the file using the bad block. Now you will know what was actually damaged.
If the
inode
number is very small and
ncheck
fails to return a path then it is probably the journal itself that is damaged. To delete the journal simply run this command on the partition:
# tune2fs -O ^has_journal /dev/sdxy
Run the
testb
command again from the
debugfs
console on the bad block and it should be no longer marked as used if it was indeed used by the journal. To build a new journal run:
# tune2fs -j /dev/sdxy
JFS
Debug the filesystem
The
jfs_debugfs
command will give you access to all the low level structures within any
JFS
filesystem. Other filesystems such as the
ext3
and
ext4
filesystems have similar tools. It is probably a good idea to
umount
any filesystem before you run this on them. To use it just run:
# jfs_debugfs /dev/sdxy
This puts you into a command console. The first thing you should note is your aggregate block size. This is (presumably) the block size the filesystem is using.
JFS
seems to default to 4096 bytes.
If you did not run
badblocks
using the block size that your filesystem is using then you will need to convert your block number(s) to match it (remember to use the block number(s) relative to the partition they are on).
i.e. block number 100 with a block size of 1024 bytes becomes block number 25 at 4096 bytes. The formula is:
(original block number) / ((filesystem block size) / (badblocks block size))
Now the entire point of running this program (for the purpose of this article) is to get the
inode
number. To do this run the command:
d
blocknumber
0 i
The syntax is the
d
command for display, the block number, the offset (just set it to 0), and the display format
i
for
inode
.
Note
If you get an error then that means the block is not allocated and is being used as free space. In that case this is a good thing as it means nothing important was damaged.
The decimal number that
di_number
is set to is the one we want. From here you type
x
to exit out of the display mode. Repeat the display command for each bad block that you have and note all of their
inode
numbers. For more info on the
inode
such as permissions and filetype type:
i
inodenumber
When you have all the
inode
numbers type
q
to quit.
Find damaged files
Finally to find the damaged file you can simply use the gnu
find
utility. Mount your filesystem and run:
# find / -inum
inodenumber
Substitute
/
for the mountpoint of the filesystem that the
inode
belongs to. If you search root and have more than one filesystem mounted you can find multiple files with the same
inode
number on different filesystems, plus
find
will take significantly longer. Remember, an
inode
is only unique to the filesystem that it is in.
XFS
Debug the filesystem
The
xfs_info
command will give you basic information about an
XFS
formatted partition and the
xfs_db
command will give you access to all the low level structures within any
XFS
filesystem.
The first thing we want to do is get the block size from the filesystem in question. Just run:
# xfs_info /dev/sdxy | grep bsize
data     =
bsize=4096
blocks=127739, imaxpct=25
naming   =version 2
bsize=4096
ascii-ci=0 ftype=1
log      =internal
bsize=4096
blocks=855, version=2
In this case 4096 is the block size being used (it appears to be the default).
If you did not run
badblocks
using the block size that your filesystem is using then you will need to convert your block number(s) to match it (remember to use the block number(s) relative to the partition they are on).
i.e. block number 100 with a block size of 1024 bytes becomes block number 25 at 4096 bytes. The formula is:
(original block number) / ((filesystem block size) / (badblocks block size))
Now the entire point of running this program (for the purpose of this article) is to get the
inode
number. Before continuing to debug it, unmount the partition. Then, run the command:
# xfs_db -c 'blockget -b
blocknumber'
/dev/sdxy
You should get output similar to this:
setting block 0/9 to data
   setting
inode to 131
for block 0/9
inode 131
block 9 at offset 0
In this example we now know the inode is 131 and can proceed to the next section
If instead you do not get an inode number in your output but do see the word
free
then likely this means that the bad block belongs to free space and nothing important was damaged.
Find damaged files
Finally to find the damaged file you can simply use the gnu
find
utility. Mount your filesystem and run:
# find / -inum
inodenumber
Substitute
/
for the mountpoint of the filesystem that the
inode
belongs to. If you search root and have more than one filesystem mounted you can find multiple files with the same
inode
number on different filesystems, plus
find
will take significantly longer. Remember, an
inode
is only unique to the filesystem that it is in.
Note
According to
xfs_db(8)
the
blockuse -n
command can be used in conjuction with
blockget -n -c
blocknumber
in order to print file names. Testing this did not produce the desired results though it may be worth trying anyway since it is faster than the find method.
# xfs_db -c 'blockget -n -b
blocknumber
' -c 'blockuse -n' /dev/sd
Xy
Force the disk to reallocate bad block
First you will want to see how many badblocks the harddrive is aware of through the
smartctl
command:
# smartctl -t long /dev/sdx
Wait until test completes, then:
# smartctl -l selftest /dev/sdx
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
To make the harddrive transparently map out the badblock with a spare good sector you will have to simply write zeros to the bad block using the
dd
command as root. Remember that with this command you have to work with the same block size as your filesystem and the block has to be relative to the partition the filesystem is on and
not
the harddrive as a whole:
# dd if=/dev/zero of=/dev/sdxy bs=4096 count=1 seek=2269012
$ sync
Alternatively,
hdparm
provides a couple of nice and simple options to read and write a given sector (4621327, in the following example):
# hdparm --read-sector 4621327 /dev/sdx
# hdparm --repair-sector 4621327 --yes-i-know-what-i-am-doing /dev/sdx
You can see if the harddrive did indeed map out an additional bad sector by checking with the
smartctl
command and seeing if the reallocated sector or event count went up:
# smartctl -A /dev/sdx
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       1
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       1
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       1
To get
Offline_Uncorrectable
to go back to 0 you need to run a SMART long test and a selftest:
# smartctl -t long /dev/sdx
Wait until test completes, then:
# smartctl -l selftest /dev/sdx
See also
Tracing a disk's sector number through md and LVM to a filesystem
by Andy Smith of BitFolk
EXT2/3 badblocks howto