Fixing fsck errors on headless system - Page 1
Forum section: electronics / general

author: nali
author_rank: Frequent Contributor
timestamp: February 05, 2025, 10:50:46 am
content: I have an Western Digital NAS which has been repurposed and now runs a few things under Debian 11 (ARM). It now has developed some filesystem errors, presumably after some multiple power outages we've had recently.Any ideas how to repair? I can't use fsck as the only interface I have is through the network, so I can't unmount the active partition. I tried /forcefsck, which does force a fsck on reboot but that still reports errors.The ext4 filesystem is black magic to me, so I've no idea how serious or fixable the errors are.Output from e2fsck after forcefsck:Code:e2fsck 1.46.2 (28-Feb-2021)Warning! /dev/sda19 is mounted.Warning: skipping journal recovery because doing a read-only filesystem check.Pass 1: Checking inodes, blocks, and sizesPass 2: Checking directory structurePass 3: Checking directory connectivityPass 4: Checking reference countsPass 5: Checking group summary informationBlock bitmap differences: +1728534 -1789536Fix? noFree blocks count wrong for group #54 (18214, counted=18213).Fix? noFree blocks count wrong (4033656, counted=4031438).Fix? noFree inodes count wrong (1568096, counted=1568074).Fix? noPadding at end of inode bitmap is not set. Fix? noSystem: ********** WARNING: Filesystem still has errors **********72064 inodes used (4.39%, out of 1640160)269 non-contiguous files (0.4%)70 non-contiguous directories (0.1%)# of inodes with ind/dind/tind blocks: 0/0/0Extent depth histogram: 66582/1062526344 blocks used (38.51%, out of 6560000)0 bad blocks1 large file56599 regular files9869 directories10 character device files1 block device file7 fifos13 links5371 symbolic links (5152 fast symbolic links)220 sockets------------72090 files

author: magic
author_rank: Super Contributor
timestamp: February 05, 2025, 10:56:19 am
content: Checking a mounted filesystem can give spurious errors but dunno if that's the case here.If in doubt, you can pull the disk out of the machine and test the unmounted filesystem elsewhere.Or see if you can boot a different OS (from a pendrive or something) on this box.

author: Psi
author_rank: Super Contributor
timestamp: February 05, 2025, 10:58:38 am
content: If you have a spare drive you could make a binary copy with DD and then fsck the copy you are not currently booted from.If you do some digging you may be able to find a linux live CD/DVD/USB that allows SSH access with default user/pass once booted, So you could ssh into the live cd and mount/fsck the drive from there.

author: nali
author_rank: Frequent Contributor
timestamp: February 05, 2025, 11:14:36 am
content: Quote from: Psi on February 05, 2025, 10:58:38 amThat's interesting, I didn't know you could fsck an image. I haven't spent too much time looking at this stage as it's still working OK, but I did look at making a Debian live boot - unfortunately they only supply images for amd64.

author: DiTBho
author_rank: Super Contributor
timestamp: February 05, 2025, 11:24:02 am
content: With kernel support, a loopback mounted image works as a block-device.

author: audiotubes
author_rank: Regular Contributor
timestamp: February 05, 2025, 03:04:28 pm
content: https://www.cyberciti.biz/faq/linux-force-fsck-on-the-next-reboot-or-boot-sequence/

author: edavid
author_rank: Super Contributor
timestamp: February 05, 2025, 03:35:24 pm
content: Quote from: nali on February 05, 2025, 11:14:36 amDon't you have any PC? You could pull the drive out of the NAS and connect it to the PC (running the live boot distribution) to do the fsck.

author: nali
author_rank: Frequent Contributor
timestamp: February 05, 2025, 04:49:17 pm
content: Quote from: audiotubes on February 05, 2025, 03:04:28 pmAre you referring to using /forcefsck (as already tried per my OP)?

author: nali
author_rank: Frequent Contributor
timestamp: February 05, 2025, 04:56:30 pm
content: Quote from: edavid on February 05, 2025, 03:35:24 pmYes, I have 2 workstations, but they are working tools and I'm disinclined to mess around with them unless I have to. The NAS is a dinky WD MyCloud Home so doesn't have pluggable drives, it'll be easier just making a boot stick with network support and going from there. The point of my post was to see if there was a trick I'm missing - seems not by the look of it.

author: edavid
author_rank: Super Contributor
timestamp: February 05, 2025, 05:15:09 pm
content: Quote from: nali on February 05, 2025, 04:56:30 pmSure it does, you just have to get the case open (which I admit is not much fun, but also not difficult).

author: audiotubes
author_rank: Regular Contributor
timestamp: February 05, 2025, 07:33:48 pm
content: Quote from: nali on February 05, 2025, 04:49:17 pmNo, I meant that you can pass options to fsck so it will fix the errors instead of taking the default of no for everything, which seems to be happening in the output you posted.

author: magic
author_rank: Super Contributor
timestamp: February 05, 2025, 07:38:13 pm
content: I don't think that it will attempt repairing a mounted filesystem, even if it's mounted read only.The only real options without moving the disk to a different machine are:1. booting another OS on the same machine2. creating an initramfs which contains fsck and booting into it with init=/bin/sh kernel parameter to run a shell instead of the usual init scripteditBut if it's headless then pretty much only 1 is viable, unless you also put ssh, network scripts and other such into the initramfs - quite a hassle.

author: audiotubes
author_rank: Regular Contributor
timestamp: February 05, 2025, 07:40:19 pm
content: I can't remember but I think the root system may be mounted R/O until some point in the boot process and then remounted R/W. And I expected that after setting the correct options to answer yes to everything, the box would be rebooted.

author: magic
author_rank: Super Contributor
timestamp: February 05, 2025, 07:43:14 pm
content: Quote from: DiTBho on February 05, 2025, 11:24:02 amIf you have an image file then you can run fsck on the file, no need for loop.But that's futile too, because an image of a filesystem mounted RW may easily be inconsistent even if the original filesystem was OK.editIf you unmount cleanly, then mount RO, then fsck the mounted filesystem or make an image and fsck the image, there should be no errors. If there are errors, something is wrong. But as soon as you mount RW, all bets are off until it's unmounted.

author: grumpydoc
author_rank: Super Contributor
timestamp: February 05, 2025, 08:32:48 pm
content: Quote from: nali on February 05, 2025, 04:56:30 pmIf the PCB looks like anything this:Then the arrow points to the serial port for the console ("TTL", not RS-232 levels), even if your board looks nothing like the above chances are there's a serial port.Though if you've got it that far apart plugging the drive into another machine using a SATA<->USB adapter becomes an option.I'm assuming the board uses GRUB to boot Linux?If you can get a console you should be able to add (might be systemd specific but AFAIK Debian 11 uses systemd) "fsck.mode=force fsck.repair=yes" to the boot command line.If you don't want to dismantle your NAS go to /boot/grub2/grub.cfg and find the command line for the kernel you are booting and add the above, then reboot.Good luck

author: nali
author_rank: Frequent Contributor
timestamp: February 06, 2025, 11:00:25 am
content: Ah well I was bored last night so decided sod it and take the thing apart. They do not want people to get into those things!Screws under stuck-on rubber bumpers. OK, that's pretty standard.Really flimsy but tightly latched internal clips holding the two halves of the case together. Even with multiple spudgers the only way to get it apart is to break clips on one side.The chassis is a snug push fit inside one half of the case. There are sprung clips hidden insideAfter all that, they kindly provide a nice convenient UART on a 0.1" header!I finally managed to fsck it.@magicguessed right earlier, the errors were false indications, file system was clean. Ah well.

author: grumpydoc
author_rank: Super Contributor
timestamp: February 06, 2025, 11:14:09 am
content: Quote from: nali on February 06, 2025, 11:00:25 amWhich brings us to the question I suppose we should have asked initially - what lead you to think there were filesystem errors in the first place?Looking back I see that fsck you ran did not replay the journal which might have explained some of the errors it reported.

author: nali
author_rank: Frequent Contributor
timestamp: February 06, 2025, 11:43:09 am
content: Quote from: grumpydoc on February 06, 2025, 11:14:09 amWe had some power issues during storm Darragh (which is not untypical round here) which resulted in intermittent power over the course of a few days which apparently was either caused or exacerbated by a substation fire. Often after a power cut we get some brief off/on events as they reroute power or whatever, so I normally turn things off until power has been back for a few hours but in this case I got caught a couple or three times.Anyway to answer your question, Samba wasn't running. Looking at the logs it was reporting insufficient inodes which I assumed was file corruption so ran fsck (which reported 5% inodes used), then went down the rabbithole. There's a good chance there was some corruption and using /forcefsck earlier acrually fixed it, I don't know.Yea, I know I should get a UPS...