[solved] catastrophic event, file recovery on ext3 - Page 1
Forum section: electronics / general

author: legacy
author_rank: Super Contributor
timestamp: October 21, 2019, 10:02:24 pm
content: a catastrophic event silently happened this morning: due to a bug in a script, a whole folder got deleted, and worse still, the script launched several resyncs before we noticed the problem.Thousands precious files lost, with Inodes overwritten by houndred of useless filesWhen the Kernel primive "file_delete" is invoked, the ext3 filesystem doesn't leave the inode pointing to a file or to a folder, but it actually zeros out the block pointers in the inode, whereas ext2 just marks these blocks as unused in the block bitmaps and marks the inode as "deleted" and leaves the block pointers alone.This means, common recovery tools (like DiskTest) simply do not work with ext3, you have to cross your finger, and try special tools, like describedhereby the first person who tried to find a solution.Code:2019-10-21--14-33-18---2019-10-21--14-35-13 - [ sys-fs/extundelete ] - success - @2.29.1/7.3.0------------------------------------------------------------------------------------------------------2019-10-21--15-21-12---2019-10-21--15-21-58 - [ sys-fs/ext3grep ] - failure - @2.29.1/7.3.0configure: error: ext3grep doesn't work on Big Endian systems------------------------------------------------------------------------------------------------------ext3grepextundeleteUnfortunatelyext3grepdoes not even compile on BIG Endian systems ( I am on a HPPA workstation ), while extundelete is less advanced, hene I moved the harddrive to a x86 machine.But ... I was not so lucky to find the folder "/home/project/DT11", probably because overwritten by the resync. At least, the inode that describes the folder "project" is gone, but other inodes might still be there.That project had backups, but the last one was made 30 days ago, and in the meanwhile my friends added a lot of new patches.I am saying, the last four weekends of work is goneSo ... I am trying searching withing blocks with the idea of grabbing some block containing the header of tarball so the block might be reversed somehow into the inode and ext3grep might try to resume the file.Each project had its tarball, kind of archive/${name}-ver${x}-rev${y}.tgz

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 21, 2019, 10:55:54 pm
content: You never back up?

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: October 21, 2019, 11:00:31 pm
content: The first thing to do is remount read-only.If the underlying storage is SSD, your data have probably beentrimmedalready.If not, then worst case, you can grep the block device for strings you know appeared in the new files, and reconstruct manually.

author: legacy
author_rank: Super Contributor
timestamp: October 21, 2019, 11:14:16 pm
content: Quote from: SiliconWizard on October 21, 2019, 10:55:54 pmThe hard drive is mirrored, but this means the content is equal to both disks. We usually save a snapshot dail, it's automatically done before midnight, and then we burn one-week-of-snapshots into a DVDRAM every Sunday afternoon; but in this case we didn't because we are still moving our storage stuff to a new place, therefore everything was manual, and the backup was planned to be monthly.We are speaking about it from the mid of September to the mid of October

author: legacy
author_rank: Super Contributor
timestamp: October 21, 2019, 11:17:58 pm
content: Quote from: HaywoodJablowme on October 21, 2019, 11:00:31 pmDone, but it was too late. The script silently did its job before someone noticed what it was doing.Quote from: HaywoodJablowme on October 21, 2019, 11:00:31 pmThe couple of disks in RAID1 are electromechanical.Quote from: HaywoodJablowme on October 21, 2019, 11:00:31 pmPrecisely what I am trying to do

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: October 21, 2019, 11:28:07 pm
content: Good luck ... you'll need it!I was successful using this method a few times. grep -C, obviously.There are also tools that scan a block device for magic such as gzip headers. This may be more efficient, if you can get it to work.

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 12:34:48 am
content: The disk is currently attached to a mac-mini/x86, running Linux.It's not mounted, only accessed as raw deviceCode:ext3grep /dev/sdb3 --search "DT11-v12-r2.tgz"Running ext3grep version 0.10.2Minimum / maximum journal block: 1545 / 35886Loading journal descriptors... sorting... doneThe oldest inode block that is still in the journal, appears to be from 1571661094 = Mon Oct 21 14:31:34 2019Journal transaction 3453486 wraps around, some data blocks might have been lost of this transaction.Number of descriptors in journal: 31738; min / max sequence numbers: 3452751 / 3453770Blocks containing "DT11-v12-r2.tgz": 3310552 4215792Code:--search str Find blocks that contain the fixed string 'str'.--search-inode blk Find inodes that refer to block 'blk'.Unfortunaly the above blocks had no good luck to find inodes that refer to them-What did you use?

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: October 22, 2019, 02:34:03 am
content: I treated the entire disk as a file*and searched not for filenames, but for plaintext file contents. (Also passed it through strings(1) first, but that's an oldskool performance optimisation, and raw grep may actually be quicker these days.) Something like:# dd if=/dev/sdb3 bs=128k | grep -C100 "some string I remember from the text file" > /tmp/sdb3_matches.txtIf you add the -b parameter to grep, it will also tell you where on the disk the matching text is.This is all super low level and also will take a long time to run (about two hours per terabyte). Ideally try other non-destructive options first.*Mantra: "everything is a stream of bytes". Repeat until enlightened. -Linus Torvalds

author: techman-001
author_rank: Frequent Contributor
timestamp: October 22, 2019, 02:44:00 am
content: Quote from: legacy on October 21, 2019, 10:02:24 pmDamn, sorry to hear that, my sympathies.I've lost data in the past from unforeseen circumstances so I feel your pain.I do hope none of the Forth stuff you were writing has been lost.

author: RoGeorge
author_rank: Super Contributor
timestamp: October 22, 2019, 03:02:08 am
content: To avoid such events for the future, ZFS filesystem can roll back changes, self repair RAID arrays and much more.

author: Whales
author_rank: Super Contributor
timestamp: October 22, 2019, 03:20:21 am
content: Eep. I know the feelings you are going through right now. Be cautious, you can easily make things worse, time spent sitting down with a cup of tea is relatively cheap in comparison.So you are saying that the metadata for your files might still exist, just spread out somewhere on the disk? Try asking for help on a relevant IRC channel, see if anyone can help. This forum isn't the best place to find kernel/filesystem experts.Otherwise: your only hope is the horrid process of piecing together blocks of raw data to get some parts of files back. Files are often not contiguous, so as file sizes increase you (exp?) have lower chance of finding a file as a single piece. Tools like photorec are fun to play with and give a graphical description of this problem (where jpegs are recovered with a reasonable chance of appearing corrupt at some point in).

author: drussell
author_rank: Super Contributor
timestamp: October 22, 2019, 03:26:50 am
content: Quote from: legacy on October 21, 2019, 10:02:24 pmPhotorecknows how to find tar headers, including the gzipped variety.If you point it at your entire disk you may be able to recover some of the ones that have not actually been overwritten (if thereareany.)

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 08:41:35 am
content: Quote from: techman-001 on October 22, 2019, 02:44:00 amyup, fortunately uForth and uLisp were on a different machine, with a copy on my PDA. But the the latest firmware is gone.Fortunately, there is a girl in my team, and you know ... women are used to taking a lot of notes, therefore we have a copy on paper about what she had modified and fixed.

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 08:56:44 am
content: Quote from: I wanted a rude username on October 22, 2019, 02:34:03 amyes, I did something similar to grab part of C files and assembly files: all partially resumed.But the real problem is how to get the whole tgz archive containing patches, documentation, etc.All the mentioned tools require you to know the inode, and I have no idea about how to get it.With the inode, and with luck, you can try to get the direct, indirect nodes pointing to data blocks.I am talking about 20Mbyte of tarball | gzipped, on a filesystem whose datablock is 4K byte of size: it means a lot (~20M/4k) of blocks sparse around the whole block device.So

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 09:13:29 am
content: Quote from: drussell on October 22, 2019, 03:26:50 amYup, I was thinking about something to recover files based on their headers, footers, and internal data structures. Sort of "file" combined with "grab". This process is commonly referred to asdata carving.PhotoRec seems based onForemost, which doens't come with a defined pattern analysis for tgz, but the toolScalpelhas it!Anyway, I tried them: no luckCode:mi2q resumed_tgz # ls09506760.tgz 12115656.tgz 12737448.tgz 13134776.tgz 14371096.tgz 14969360.tgz 25218536.tgz 30216704.tgz10152855.tgz 12115768.tgz 12781008.tgz 13140864.tgz 14379160.tgz 14969920.tgz 25424944.tgz 30219264.tgz10152865.tgz 12115952.tgz 12793184.tgz 13145016.tgz 14404592.tgz 14975712.tgz 25501688.tgz 30517472.tgz10953552.tgz 12116064.tgz 12793296.tgz 13145144.tgz 14422032.tgz 14975960.tgz 25514160.tgz 30585584.tgz10953960.tgz 12117600.tgz 12793936.tgz 13168200.tgz 14422064.tgz 14983272.tgz 25911848.tgz 30585624.tgz10954024.tgz 12127760.tgz 12793960.tgz 13173048.tgz 14429496.tgz 14983304.tgz 26810464.tgz 30585680.tgz10959928.tgz 12127872.tgz 12822856.tgz 13176160.tgz 14443064.tgz 14985280.tgz 26810816.tgz 30917376.tgz11003920.tgz 12128136.tgz 12823816.tgz 13191216.tgz 14445176.tgz 14985776.tgz 26811400.tgz 31672944.tgz11003984.tgz 12128376.tgz 12824008.tgz 13193008.tgz 14473360.tgz 15050808.tgz 26828256.tgz 31973480.tgz11004472.tgz 12178408.tgz 12842872.tgz 13202656.tgz 14474856.tgz 15050840.tgz 26859136.tgz 31973776.tgz11004720.tgz 12214216.tgz 12844416.tgz 13226392.tgz 14480456.tgz 15053824.tgz 26860096.tgz 31973936.tgz11005048.tgz 12214240.tgz 12878272.tgz 13253704.tgz 14497008.tgz 15065088.tgz 26861120.tgz 31974104.tgz11008800.tgz 12215200.tgz 12892440.tgz 13257312.tgz 14502912.tgz 15074776.tgz 26907552.tgz 31975272.tgz11014696.tgz 12237608.tgz 12892552.tgz 13278584.tgz 14503304.tgz 15074808.tgz 26925080.tgz 31977392.tgz11017720.tgz 12238704.tgz 12893136.tgz 13309800.tgz 14521384.tgz 15082456.tgz 26925400.tgz 32156760.tgz11019304.tgz 12264064.tgz 12893200.tgz 13346128.tgz 14528528.tgz 15088328.tgz 26925656.tgz 32157520.tgz11019664.tgz 12264176.tgz 12893312.tgz 13387248.tgz 14528704.tgz 15088352.tgz 26925704.tgz 32169808.tgz11019856.tgz 12265304.tgz 12899288.tgz 13402472.tgz 14528736.tgz 15089696.tgz 26925720.tgz 32875048.tgz11036032.tgz 12269328.tgz 12901656.tgz 13500424.tgz 14531616.tgz 15089992.tgz 26925976.tgz 32875608.tgz11036912.tgz 12279504.tgz 12901752.tgz 13533984.tgz 14531624.tgz 15091032.tgz 26926064.tgz 32875720.tgz11087600.tgz 12307440.tgz 12905376.tgz 13697072.tgz 14589952.tgz 15091512.tgz 26926192.tgz 32922264.tgz11087736.tgz 12324880.tgz 12913696.tgz 13753096.tgz 14591400.tgz 15094136.tgz 26926320.tgz 34556280.tgz11107296.tgz 12324904.tgz 12914336.tgz 13753128.tgz 14596968.tgz 15140232.tgz 27087368.tgz 35274120.tgz11107408.tgz 12364560.tgz 12914912.tgz 13793424.tgz 14610016.tgz 15140672.tgz 27590720.tgz 36124224.tgz11107856.tgz 12368992.tgz 12930200.tgz 13800064.tgz 14653816.tgz 15142496.tgz 27591280.tgz 36132736.tgz11144448.tgz 12369040.tgz 12962144.tgz 13802328.tgz 14678856.tgz 15145936.tgz 27593784.tgz 36135656.tgz11192576.tgz 12369400.tgz 12966176.tgz 13803056.tgz 14679688.tgz 15147656.tgz 27597448.tgz 37041488.tgz11203656.tgz 12405472.tgz 12966288.tgz 13964280.tgz 14687224.tgz 15148512.tgz 27597576.tgz 37444912.tgz11276304.tgz 12405784.tgz 12966312.tgz 13964312.tgz 14716616.tgz 15149960.tgz 27772584.tgz 37595856.tgz11336408.tgz 12406232.tgz 12967064.tgz 13964512.tgz 14716648.tgz 15151904.tgz 27772616.tgz 37599360.tgz11436040.tgz 12406680.tgz 12969016.tgz 13966152.tgz 14726344.tgz 15152096.tgz 27774048.tgz 37617360.tgz11549512.tgz 12410992.tgz 12973000.tgz 13967672.tgz 14729736.tgz 15152432.tgz 28345912.tgz 37618920.tgz11549624.tgz 12491720.tgz 12999128.tgz 13968256.tgz 14733000.tgz 15162216.tgz 28461032.tgz 37629456.tgz11659040.tgz 12495576.tgz 12999384.tgz 13969088.tgz 14778624.tgz 15166152.tgz 28461536.tgz 37646632.tgz11661024.tgz 12495600.tgz 12999472.tgz 13970272.tgz 14778656.tgz 15169424.tgz 28465496.tgz 37669640.tgz11661048.tgz 12496096.tgz 12999568.tgz 13972912.tgz 14779816.tgz 15173192.tgz 28466640.tgz 37674344.tgz11662832.tgz 12506040.tgz 13000248.tgz 13975120.tgz 14780352.tgz 15173400.tgz 28467144.tgz 38024024.tgz11822680.tgz 12535776.tgz 13001304.tgz 13975464.tgz 14783648.tgz 15173648.tgz 28532472.tgz 38917432.tgz11844048.tgz 12535824.tgz 13015920.tgz 14069072.tgz 14786024.tgz 15174736.tgz 28533653.tgz 38917616.tgz11844088.tgz 12599704.tgz 13016720.tgz 14069104.tgz 14786648.tgz 15175928.tgz 28535973.tgz 39415904.tgz11844344.tgz 12599816.tgz 13020136.tgz 14071424.tgz 14796440.tgz 15195520.tgz 28542256.tgz 39570672.tgz11868384.tgz 12600496.tgz 13020168.tgz 14094680.tgz 14797480.tgz 15197296.tgz 28542624.tgz 39626720.tgz11885624.tgz 12601920.tgz 13020680.tgz 14095392.tgz 14798088.tgz 15197592.tgz 28543712.tgz 40094032.tgz11895688.tgz 12621656.tgz 13022216.tgz 14095752.tgz 14799832.tgz 15199968.tgz 28790640.tgz 40094384.tgz11895800.tgz 12629320.tgz 13025496.tgz 14098560.tgz 14800680.tgz 15201384.tgz 29022120.tgz 40631362.tgz11895824.tgz 12631368.tgz 13030496.tgz 14116024.tgz 14802280.tgz 15208648.tgz 29026112.tgz 42173504.tgz11905904.tgz 12632328.tgz 13038760.tgz 14116744.tgz 14837008.tgz 15211904.tgz 29026120.tgz 42813432.tgz11915272.tgz 12632352.tgz 13044680.tgz 14140384.tgz 14837040.tgz 15212400.tgz 29321488.tgz 43116816.tgz11918904.tgz 12632688.tgz 13044944.tgz 14256096.tgz 14838632.tgz 15213616.tgz 29321728.tgz 43848776.tgz11933072.tgz 12632800.tgz 13045016.tgz 14256152.tgz 14839488.tgz 15233464.tgz 29321976.tgz 43922464.tgz11933208.tgz 12635264.tgz 13045344.tgz 14256272.tgz 14840616.tgz 15237800.tgz 29322224.tgz 43924256.tgz11952016.tgz 12662808.tgz 13058936.tgz 14257200.tgz 14840648.tgz 15240848.tgz 29322344.tgz 44101832.tgz11993192.tgz 12662832.tgz 13090912.tgz 14259200.tgz 14841840.tgz 15284136.tgz 29322616.tgz 44276472.tgz12003536.tgz 12667416.tgz 13092656.tgz 14259360.tgz 14880976.tgz 15301216.tgz 29323120.tgz 44863504.tgz12045704.tgz 12667992.tgz 13092704.tgz 14259544.tgz 14909200.tgz 16203928.tgz 29323536.tgz 44863536.tgz12045816.tgz 12668432.tgz 13093080.tgz 14259800.tgz 14909872.tgz 22935264.tgz 29323944.tgz 44863952.tgz12046144.tgz 12669936.tgz 13095320.tgz 14275152.tgz 14911160.tgz 23929496.tgz 29327792.tgz12046168.tgz 12669960.tgz 13095456.tgz 14275368.tgz 14914024.tgz 24261726.tgz 29717482.tgz12049392.tgz 12670048.tgz 13120712.tgz 14295008.tgz 14939112.tgz 24274846.tgz 30200904.tgz12083312.tgz 12703408.tgz 13122936.tgz 14297088.tgz 14939568.tgz 24519358.tgz 30204416.tgz12083400.tgz 12733936.tgz 13123000.tgz 14365832.tgz 14947088.tgz 24521778.tgz 30206536.tgz12083424.tgz 12735496.tgz 13123408.tgz 14370048.tgz 14963608.tgz 24522970.tgz 30209848.tgz12101320.tgz 12736320.tgz 13128016.tgz 14370080.tgz 14968488.tgz 25213160.tgz 30214208.tgzThe 90% of these files is a corrupted tgz, and the 10% of them don't contain what I was looking for, but it contains a very old backup, that was deleted time ago.And this is what I do not understand: the data carving stuff is able to find and resume tgz files made 2 months ago, and deleted 1 month ago, but it's not able to find and resume a tgz that was created 2 days ago and deleted yesterday.It should have a high probability to have been overwritten, as well as recent deleted files should have a low probability to have been overwrittenSomething tells me the LAST algorithm is not used for ext3, which, if it's true, will make me abandon this filesystem because it's worrying.Combined with these annoying things:Code:EXT4-fs (sda3): couldn't mount as ext3 due to feature incompatibilitieswhich I still have to understand ...

author: BravoV
author_rank: Super Contributor
timestamp: October 22, 2019, 09:23:27 am
content: Damn .. I'm in the phase of preparing to move my whole rig to full Linux as host from Windows, as Linux noob, you guys are scaring me.

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 09:25:46 am
content: Quote from: techman-001 on October 22, 2019, 02:44:00 amThanks for this. Fresh news, my partner is telling me just now that the problem was caused not by a bug in a script but rather by a bug in the bash shell.Which explains what really happened: the env variable about the project's root was not evaluated at all, pointing rm -r to the /home/project rather than to /home/project/${var}This symptom silently and randomly happens on our new stage4 after having recompiled the core system, which is planned weekly at the beginning of the week.Yesterday, it was Monday(I seriously hates Gentoo for this), it seems the last stage4 for MIPS and HPPA is affected, so updating bash, python and perl (and probably other stuff) was not a good idea.I have the backup of the previous stage4. I am going to buy a new hard drive, and to restore it blocking any update until the bash scripting engine is 100% sure working, keeping this harddrive here in read-only for data resuming purpose.

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: October 22, 2019, 10:26:40 am
content: Gentoo in production? You're a braver man than I.Newer files being more likely to be corrupted may be due to fragmentation. Those recovery tools usually can only recover contiguous files. If the file spans more than one contiguous extent, they won't even have a good idea of where it ends.Have you tried running the more promising looking files through gzrt to see if at least the first part of them is recoverable?

author: techman-001
author_rank: Frequent Contributor
timestamp: October 22, 2019, 10:32:59 am
content: Quote from: RoGeorge on October 22, 2019, 03:02:08 amAgreed and probably why I haven't lost any data in the last three years.About a year ago I made the incredibly stupid mistake of setting *all* my Workstation /home file attributes to read/write only, about 4GB of data with hundreds of directories. I had a brief WTF moment then remembered I had snapshots of /home running and historical snapshots dating back the previous 2 years (zfstools). No gap in snapshot data exceeding 15 minutes meant I I would only lose data to a maximum of 15 minutes age.About 5 minutes later everything was fixed.

author: TK
author_rank: Super Contributor
timestamp: October 22, 2019, 10:51:24 am
content: If you rewrite the code starting from the 30-day old backup, you can finish it in 2 weeks with a code that will be much better than the original one that was lost. I suggest you take this route and not waste more time trying to recover the files from the maze that is going to be tracking the blocks and hoping for a partial recovery

author: AndyC_772
author_rank: Super Contributor
timestamp: October 22, 2019, 11:23:06 am
content: Quote from: legacy on October 21, 2019, 10:02:24 pmDo they not have copies of the changes they made?

author: sokoloff
author_rank: Super Contributor
timestamp: October 22, 2019, 11:41:18 am
content: Quote from: legacy on October 21, 2019, 11:14:16 pmThis is why “RAID is not a backup!” is a sysadmin aphorism. Some levels of RAID help data availability; others help access speed; none are backups.If others have git repo copies, you should be able to recover most of the content. If they otherwise have working copies, you may be able to recover partially. If they have backups (TimeMachine or other?), you may have partial data around somewhere.Otherwise, I like the “rewriting is often faster and better anyway” outlook.

author: magic
author_rank: Super Contributor
timestamp: October 22, 2019, 12:02:22 pm
content: Quote from: legacy on October 21, 2019, 10:02:24 pmI believe ext3 journals writes to directory data blocks so full information about older revisions of this directory should be found in the journal.So start with reviewing older revisions of /home inode. See what directory data blocks it points to (may be the same blocks as the current revision of /home). Find older revisions of those blocks in the journal. They will point you to your file inodes.Of course no guarantee that file data are not overwritten.

author: sleemanj
author_rank: Super Contributor
timestamp: October 22, 2019, 12:06:49 pm
content: Ouch. I can't offer any remedial help that has not already been, but for future, borg backup is good, chunk de-duped compressed incremental backups very efficient on space and pretty fast, mounts backups as userfs for easy restore, and is easy to use.Good luck with the recovery, worst case, at least now you know all the mistakes you made in the last month so maybe it only takes you 3 weeks re-coding :-/

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 12:25:24 pm
content: ZFS filesystem does not compile on HPPA, MIPS, HPPA, and it consumes too much ram.It's no go.BTRFS, may be, but for now we have reason to say with ext3

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 12:26:51 pm
content: Quote from: sleemanj on October 22, 2019, 12:06:49 pmYup, written notes on paper have survived, and this helps

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 12:31:33 pm
content: Quote from: I wanted a rude username link=topic=215246.msg2749520#msg2749520yup, this way I recovered C files and assembly files. Small files, whose blocks have low fragmentation.

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 12:40:02 pm
content: Quote from: I wanted a rude username on October 22, 2019, 10:26:40 amGentoo in production and on experimental platforms.The HPPA one is more stable than the MIPS4 one; anyway ... I have never thought a bash shell can behave this way, the whole /etc/system/* relies on bash scripts, so if the bash goes crazy after a recompile, this is really really worrying.Lesson learned: never trust a new bash shell before you have run ALL the tests. This stuff needs to stay in quarantine inside a sandbox until you are 100% sure it's safe.

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 22, 2019, 01:18:23 pm
content: Quote from: legacy on October 21, 2019, 11:14:16 pmMurphy's law rarely fails to hold true.I'm sure during the time you had daily backups, you probably hardly encountered any failure?And now that you temporarily switched to a monthly backup, you have a fuck-up 30 days right after the last backup? That's Murphy almost to the letter!I understand the temporary situation, but going from daily to monthly was a bit on the edge anyway...

author: SiliconWizard
author_rank: Super Contributor
timestamp: October 22, 2019, 04:00:53 pm
content: Just remembering one event... when it comes to fuck-ups...Not too long ago, I issued a "rm -rf /boot" as root on one of my machines.This was meant to be "rm -rf boot" (removing a local boot directory, but with root ownership), but "autopilot" plays tricks when you're used to typing '/boot' much more often than just 'boot'.I had no recent full system backup, but I had a copy of all boot config files, so rebuilding the /boot directory was just a matter of a couple minutes, no harm done. But this is still one of those times when you are in awe for a few seconds for doing such a stupid thing...

author: sokoloff
author_rank: Super Contributor
timestamp: October 22, 2019, 06:24:44 pm
content: I had an nfs volume mounted with no_root_squash and went to rm -rf the local copy of all home directories that I'd copied over to the nfs volume so I could re-create the mount point and mount user dirs over nfs.Unfortunately, I picked the wrong target and rather than quickly deleting the modest size home directory of a couple local users, it began to slowly rm -rf all home dirs on the nfs volume. I gave it a few seconds to complete before the reality set in of what I'd just done. Also no backups...

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 06:50:52 pm
content: I wonder which is the behavior, in this case, of RaiserFS and XFS.Do they use the LAST(1) policy(2) for choosing the "free" inode on file creation?Do they zero structures on file and folder deletion?I will check it out.(1) this means that if you delete a folder, and you do not issue "sync", its inode lists is still in ramand will be reused for new files, which is cathastropic because everything you have just deleted is going to be overwrittenToday I have resumed a lot of files, and LAST is what I have observed for most of the data truncated/overwritten.(2) can this policy be changed? I have to investigate

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: October 22, 2019, 08:56:41 pm
content: This all becomes moot with SSDs, anyway. We can't rely on implementation quirks to secure our data. A combination of online and offline backup methods, with good reporting by email (with ausefulsummary in the subject field), is the only way to be sure.

author: james_s
author_rank: Super Contributor
timestamp: October 22, 2019, 09:41:23 pm
content: Quote from: sokoloff on October 22, 2019, 06:24:44 pmWhoops.I once did sudo rm -rf and hit enter before I'd typed the full path on my work PC and blew away the whole bin folder which had all the system files like cp and whatnot. I ended up having to boot up from a live cd of the same distro and copy it over from that. Lesson learned.

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 09:42:16 pm
content: I do think SSD is no matter unless you modify the kernel driver. A SSD disk is managed as a random block device, the kernel addresses it by LBAs, and LBAs are chosen by LAST and its policy, therefore if you format it ext3 you have the same behavior as on an electromechanical disk.In fact, there are two patches for SSDs, and they work at the low level (SATA controller), and they won't work on HPPA and MIPS, this due to the nature of x86 SATA controllers used in non-x86 computers, which are "too much modern " stuff for the technology, and rather "snubbed" by the industry, in fact, there are no working hw-raid cards, only "hacked" sw-raid controllers. You can imagine the ZERO support for SSD-optimized SATA controllers, which are also only for PCIexpress, while we are still on PCI-X and PCI64.My team spent weeks checking and testing PCI SATA controllers, signaling and fixing bugs, and I know, we'd better avoid SSDs; I do believe SSDs can have the same "shuffle" effect unless you seriously activate SSD specific support in the kernel.Anyway, now the point is: to stay with ext3, or to move to XFS or RaiserFS?Backups will be planned as usual: weekly!

author: sokoloff
author_rank: Super Contributor
timestamp: October 22, 2019, 10:19:26 pm
content: HPPA, as in the pre-Itanium HP-PA?That HP stopped selling about 10 years ago and dropped support for in datacenters in 2013?In a case like that, you might want to stick with ext3 and backups.If you're stuck on that platform, you might consider ISCSI or nfs targets hosted on a more modern platform.I ran reiserfs for years, lost a few disks but never lost any data. When the Hans Reiser case broke and he was arrested, I figured that was the death of reiserfs as far as long term supportability went.I switched to zfs about 2 years ago at home (first on FreeNAS, then on bare Linux). I like it and I'm happy to throw RAM at it for the security and performance (and hands-off ease, frankly) that it gives me. I ship backups locally with zfs send/receive and remotely a subset of mount points are backed to S3/glacier.

author: legacy
author_rank: Super Contributor
timestamp: October 22, 2019, 10:37:08 pm
content: Quote from: sokoloff on October 22, 2019, 10:19:26 pmYup. Even if it was announced decommissioning in 2008, it last until 2013.But, this stuff had HPUX, while Linux has never been stable, neither supported.So, in this combination, you like it only if you likewildstuffQuote from: sokoloff on October 22, 2019, 10:19:26 pmYup, iSCSI is in our plansBut first, we have to fix the evil Bash-shell which caused this mess.

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: October 22, 2019, 11:58:41 pm
content: Quote from: legacy on October 22, 2019, 09:42:16 pmNot for deletion. If Linux detects that a device is an SSD it will periodicallytrimit (send it TRIM/UNMAP commands for the deleted ranges), causingthe device itselfto erase deleted extents. This is the default behaviour ... and you probably don't want to disable it, because then performance degrades (forcing erase-before-write mode).If that disk were an SSD, you would not be finding any old deleted files on it now.

author: legacy
author_rank: Super Contributor
timestamp: October 23, 2019, 03:36:25 am
content: Quote from: I wanted a rude username link=topic=215246.msg2750384#msg2750384"If it detects" means the kernel driver needs specific support, and this is precisely what does not work with x86 sata controllers on non-x86 machines(1)This usually needs some "bios_setup" part made in the firmware, and it's critical because the Linux kernel expects the initialization of the controller already done by the BIOS, with a couple of BIOS_extension parts written in x86 and installed inside the controller into the ROM. All of these things suck on HPPA and MIPS, so SSDs are managed in "compatibility mode".(1) perhaps, except ARM and POWER9. The last one mostly because it milks money from DARPA, hence it might have this things worked around, somehow ... but it seems they prefer AIX.edit: when we ask to THE people with skills, the typical answer isQuoteHere you can understand why non-x86's there is so little support: not because there are technical problems, but rather because hobbysts do not have enough skills to fix the stuff, while THE people with competence want money, a lot of money.In short, because "if you are good at something, never do it for free"

author: legacy
author_rank: Super Contributor
timestamp: October 23, 2019, 03:48:55 am
content: Quote from: I wanted a rude username on October 22, 2019, 11:58:41 pmYup, and it's one of the reasons why we are dealing with electromechanical hard drives. SSDs downgraded into "compatibility mode" have a very bad performance response, hence they are not worth the money, compared to electromechanical hard drives.non-x86 workstations have a lot of limitations, especially with Linux.

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: October 23, 2019, 04:07:59 am
content: Quote from: legacy on October 23, 2019, 03:36:25 amAnd I thoughtIwas a masochist. True though, there just isn't the open source community support for workstations. Alphas were the same, back in the day.Hopefully RISC-V based computers will fill the "powerful, popular, and not x86" gap in the next decade ... but they won't give us that nice big-endian flavour.

author: Whales
author_rank: Super Contributor
timestamp: October 23, 2019, 11:01:35 am
content: Quote from: I wanted a rude username on October 22, 2019, 11:58:41 pmN.B. I do not believe this is entirely true.Linux won't auto-trim unless the filesystem is mounted with 'discard' option (and uses a filesystem that supports this feature, ext3 isn't one IIRC). Some distro install/setup scripts may add this mount option to fstab automatically and/or schedule manual fstrim runs in userspace instead (eg via cronjob or systemd-timer-lizard thingy).Otherwise: TRIM is a completely opt-in procedure.

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: October 23, 2019, 11:55:48 am
content: Quote from: Whales on October 23, 2019, 11:01:35 amYes ... perhaps I should have said "GNU/Linux".Code:$ systemctl status fstrim.timer● fstrim.timer - Discard unused blocks once a weekLoaded: loaded (/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)Active: active (waiting) since Thu 2019-09-19 16:56:48 AEST; 1 months 3 days agoTrigger: Mon 2019-10-28 00:00:00 AEDT; 4 days leftDocs: man:fstrim

author: Whales
author_rank: Super Contributor
timestamp: October 23, 2019, 08:51:03 pm
content: Quote from: I wanted a rude username on October 23, 2019, 11:55:48 amThanks for the job snippet -- I would have thought they would schedule it daily, but I guess it doesn't matter too much. A weekshouldgive you enough headroom to notice a data-loss problem and take things offline.QuoteNo no, it's systemd/Linux these days

author: I wanted a rude username
author_rank: Frequent Contributor
timestamp: October 23, 2019, 09:11:33 pm
content: Quote from: Whales on October 23, 2019, 08:51:03 pmThis physically hurt to read.

author: legacy
author_rank: Super Contributor
timestamp: October 28, 2019, 12:14:24 pm
content: Quote from: magic on October 22, 2019, 12:02:22 pmIn parallel, we cloned the harddrive for a second mac-mini/x86 which is still processing.sort of for inode={2 to ....}, process(inode)So, this approach is very very slow.

author: legacy
author_rank: Super Contributor
timestamp: October 28, 2019, 12:25:19 pm
content: Two tools have been developed during this story:a text viewer (similar to hexedit, but *text* and) able to open a giant file of 40GByte, processing data only if they look "printable" "strings"a pattern matching searcher (similar to grep but) able to work within a region (given start address and end address regarding the seek position in the input big file, and able to) grab stuff when it sees a matching pattern, and stop grabbing when it sees a stop patternexamplegrab between 0x410FAD9A1 and 0x0x610FAD9A1, everything that begins with "/*" untill you see '\0'This generated some local files, mostly are false positive, but this way it extracted the 90% of lost C files.

author: legacy
author_rank: Super Contributor
timestamp: October 28, 2019, 12:45:50 pm
content: Thank everyone for the support. It's much more than appreciated

author: StillTrying
author_rank: Super Contributor
timestamp: October 28, 2019, 01:49:35 pm
content: You're lucky there legacy, I was just about to quote your troll and log post.It took me ages to decide where it was from, and the timestamps were out of order.