The strange ways a hard drive can die - Page 1
Forum section: electronics / general

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2025, 12:34:38 pm
content: During my years of computer use, It only happens a few times that an hdd suddenly died, and it happened in different ways ...Opening it for a sort of "biopsy", I found the a couple of hdds died due to a high vibrational shock which crushed the heads down to the spinning plat so hard that they scratched.In particular, I remember a guy fell on me while I was on the subway carrying my laptop in a bag, and it hit one of the steel poles on the carriage. The plastic also cracked. It was a pretty energetic impact.One HDD died due to excessive temperature or something burning out the circuit board. Bad airflow is a bad idea. I know this because after replacing the circuit board with a new one, the HDD started working again.-Yesterday I was resyncing my files to the NAS, which schedules backups on DVDRAM, DDS4 and LTO2.During the network transfer from my mac-mini intelcore2 to the NAS something weird happened, something that damaged the internal hdd, but in a way that I have never experimented.Code:lelly idp # cp -r lib_dependency_management_v1Display all 107 possibilities? (y or n)lelly idp # cp -r lib_dependency_management_v1 /mnt/disk2/Xilinx-ISE-v14.7-1015-1-Linux.md5 Xilinx-ISE-v14.7-1015-1-Linux.tar golang/ lost+found/ macmini-i2-disaster-2025-08-30/lelly idp # cp -r lib_dependency_management_v1 /mnt/disk2/macmini-i2-disaster-2025-08-30/cp: cannot stat `lib_dependency_management_v1': Input/output errorlelly idp # cp -r lib_dependency_management_v1 /mnt/disk2/macmini-i2-disaster-2025-08-30/cp: cannot stat `lib_dependency_management_v1': Input/output errorlelly idp # cd lib_dependency_management_v1Display all 107 possibilities? (y or n)lelly idp # cd lib_dependency_management_v1bash: cd: lib_dependency_management_v1: Input/output errorlelly idp # cd lib_dependency_management_v1lelly lib_dependency_management_v1 # lsMakefile lib_dependency_management_not_recursive_mini_v1.c lib_fifo_v1 project.ilib.myprjapp lib_dependency_management_not_recursive_mini_v1.h lib_fifo_v1___specific_for_this_project___to_be_renamed project.lib.myprjarchive lib_dependency_management_not_recursive_v1.c lib_file_query_v1 project.list.myprjbug.txt lib_dependency_management_not_recursive_v1.h lib_file_query_v1___specific_for_this_project___to_be_renamed project.name.myprjdoc lib_dependency_management_recursive_v1.c lib_linea_fd_v1 project.root.myprjguineapig lib_dependency_management_recursive_v1.h log2.txt project.run.myprjlib_dependency_management_core_v1.c lib_dependency_management_v1.c out researchinglib_dependency_management_core_v1.h lib_dependency_management_v1.h project.geanylelly lib_dependency_management_v1 # mkdir /mnt/disk2/macmini-i2-disaster-2025-08-30/lib_dependency_management_v1_recoveredlelly lib_dependency_management_v1 # cp -r * /mnt/disk2/macmini-i2-disaster-2025-08-30/lib_dependency_management_v1_recovered/cp: cannot stat `lib_file_query_v1/lib_file_query_v1.h.gch': Input/output errorcp: cannot stat `out/macmini2-intel-x86_64/lib_dependency_management_v1.o': Input/output errorcp: cannot stat `out/desc/lib_dependency_management_not_recursive_mini_v1.private': Input/output errorcp: cannot stat `out/desc/lib_dependency_management_core_v1.private': Input/output errorcp: cannot stat `out/desc/lib_dependency_management_not_recursive_v1.private': Input/output errorlelly lib_dependency_management_v1 #I talked about this library here on the forum, it's a complex library, of course I have a backup, but it's not point now, the point is ... look at the errors ... look at how the hdd behavesCode:xfs-fs error (device sdb1): __xfs_get_inode_loc:4729: inode #15501198: block 61868184: comm bash: unable to read itable blocksd 1:0:0:0:  Unhandled sense codesd 1:0:0:0:  Result: hostbyte=0x10 driverbyte=0x08sd 1:0:0:0:  Sense Key : 0x3 sd 1:0:0:0:  ASC=0x11 ASCQ=0x0sd 1:0:0:0:  CDB: cdb=0x28: 28 00 1d bc 14 bc 00 00 08 00end_request: critical target error, dev sdb, sector 498865340sd 1:0:0:0:  Unhandled sense codesd 1:0:0:0:  Result: hostbyte=0x10 driverbyte=0x08sd 1:0:0:0:  Sense Key : 0x3 sd 1:0:0:0:  ASC=0x11 ASCQ=0x0sd 1:0:0:0:  CDB: cdb=0x28: 28 00 1d bc 14 bc 00 00 08 00end_request: critical target error, dev sdb, sector 498865340xfs-fs error (device sdb1): __xfs_get_inode_loc:4729: inode #15501201: block 61868185: comm cp: unable to read itable blocksd 1:0:0:0:  Unhandled sense codesd 1:0:0:0:  Result: hostbyte=0x10 driverbyte=0x08sd 1:0:0:0:  Sense Key : 0x3 sd 1:0:0:0:  ASC=0x11 ASCQ=0x0sd 1:0:0:0:  CDB: cdb=0x28: 28 00 1d bc 14 bc 00 00 08 00end_request: critical target error, dev sdb, sector 498865340xfs-fs error (device sdb1): __xfs_get_inode_loc:4729: inode #15501206: block 61868185: comm cp: unable to read itable blocksd 1:0:0:0:  Unhandled sense codesd 1:0:0:0:  Result: hostbyte=0x10 driverbyte=0x08sd 1:0:0:0:  Sense Key : 0x3 sd 1:0:0:0:  ASC=0x11 ASCQ=0x0sd 1:0:0:0:  CDB: cdb=0x28: 28 00 1d bc 14 bc 00 00 08 00end_request: critical target error, dev sdb, sector 498865340xfs-fs error (device sdb1): __xfs_get_inode_loc:4729: inode #15501211: block 61868185: comm cp: unable to read itable blocksd 1:0:0:0:  Unhandled sense codesd 1:0:0:0:  Result: hostbyte=0x10 driverbyte=0x08sd 1:0:0:0:  Sense Key : 0x3 sd 1:0:0:0:  ASC=0x11 ASCQ=0x0sd 1:0:0:0:  CDB: cdb=0x28: 28 00 1d bc 14 bc 00 00 08 00end_request: critical target error, dev sdb, sector 498865340xfs-fs error (device sdb1): __xfs_get_inode_loc:4729: inode #15501210: block 61868185: comm cp: unable to read itable blocksd 1:0:0:0:  Unhandled sense codesd 1:0:0:0:  Result: hostbyte=0x10 driverbyte=0x08sd 1:0:0:0:  Sense Key : 0x3 sd 1:0:0:0:  ASC=0x11 ASCQ=0x0sd 1:0:0:0:  CDB: cdb=0x28: 28 00 1d bc 14 bc 00 00 08 00end_request: critical target error, dev sdb, sector 498865340xfs-fs error (device sdb1): __xfs_get_inode_loc:4729: inode #15501212: block 61868185: comm cp: unable to read itable blockit's crazy, it's like nothing I've ever seen, it's different.

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2025, 12:38:07 pm
content: Code:lelly lib_dependency_management_v1 # ls /mnt/disk2/macmini-i2-disaster-2025-08-30/lib_dependency_management_v1_recovered/Makefile lib_dependency_management_not_recursive_mini_v1.c lib_fifo_v1 project.ilib.myprjapp lib_dependency_management_not_recursive_mini_v1.h lib_fifo_v1___specific_for_this_project___to_be_renamed project.lib.myprjarchive lib_dependency_management_not_recursive_v1.c lib_file_query_v1 project.list.myprjbug.txt lib_dependency_management_not_recursive_v1.h lib_file_query_v1___specific_for_this_project___to_be_renamed project.name.myprjdoc lib_dependency_management_recursive_v1.c lib_linea_fd_v1 project.root.myprjguineapig lib_dependency_management_recursive_v1.h log2.txt project.run.myprjlib_dependency_management_core_v1.c lib_dependency_management_v1.c out researchinglib_dependency_management_core_v1.h lib_dependency_management_v1.h project.geanyoh, but see ... in the end, it copied part of the files, and they are not corrupted

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2025, 12:45:27 pm
content: While after an hard shock, there were sectors that were unreadable, and there was no way to read them, because I was in a bad cluster, here we have sectors that, if you insist, you can read them... except that the kernel reports a lot of "sensing" errors, as if the heads were struggling, or there was something interfering.Can't be dust! An HDD has a dust-filter!It almost seems like there's something on the platters, like some stains ... the HDD would have to be opened to understand what happened.Anyway, the disk behaved like this suddenly, it wasn't subjected to any shocks, it has always operated at a controlled humidity and temperature, at 24°C, and this anomalous behavior manifested itself during a resync read.____W____T____F____

author: Shock
author_rank: Super Contributor
timestamp: August 31, 2025, 01:06:58 pm
content: Quote from: DiTBho on August 31, 2025, 12:45:27 pmThat is exactly what I thought when I read it.

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2025, 01:11:13 pm
content: Quote from: Shock on August 31, 2025, 01:06:58 pmRead what?a. The error message reported by the kernel?b. The strange behavior of insisting on reading file to have a better chance of success?c. ...?

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2025, 01:29:13 pm
content: In the end, I didn't lose any development files (libraries, C stuff, and HDL) because they were backuped months ago, but I completely lost my Stage4 because I hadn't backed it up.Working without backups is a always bad idea, I know, I know, but I painted my house and did some work, so I physically shut down and dismantled some computers, including the server that runs the backups.Classic mistake... "What could possibly go wrong during the summer months?"I have to buy a new hard driveworse still, I have to open the mac-mini/2009 to install it ... I hate this partTwo months of weekend work have gone to waste. I have to download a Stage3 and start over with gnat, gcc/crosscompilers&C, crossdev, tons of patches ... usual Gentoo stuff*but* if the damage was caused by a spike on the 230Vac line, I'd betterbuy an UPSwith a nice noice-filter

author: Gyro
author_rank: Super Contributor
timestamp: August 31, 2025, 02:06:02 pm
content: Quote from: DiTBho on August 31, 2025, 12:45:27 pmThe heads can get covered in all sorts of sh*t after a crash, not to mention distortions in the suspension. Also, the impact area on the platter doesn't have smooth edges so the head can continue to stir up fresh debris as it cross it.

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2025, 02:16:20 pm
content: Quote from: Gyro on August 31, 2025, 02:06:02 pmyes, but in *this* case there was not a crash!The hdd worked inside the Mac-mini, in an environment that is free of vibrations and is controlled in humidity and temperature.Suddenly, it manifested weird errors during file operations.

author: magic
author_rank: Super Contributor
timestamp: August 31, 2025, 02:58:36 pm
content: I guess you won't get far without finding what this SCSI mumbo-jumbo means and reading out SMART.

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2025, 03:08:14 pm
content: Quote from: magic on August 31, 2025, 02:58:36 pmA mac-mini/2009 uses sATA 2.5" disks. It can have two if you remove the optical drive and buy a carrier.At the moment it's attached to a USB-sATA adapter.It seems S.M.A.R.T. doesn't directly operate on itCode:lelly ~ # smartctl /dev/sdbsmartctl 5.39.1 2010-01-28 r3054 [powerpc-unknown-linux-gnu]/dev/sdb: Unknown USB bridge [0x152d:0xa578 (0x214)]Smartctl: please specify device type with the -d option.I don't have an other computer here with a sATA adapter.I don't know what S.M.A.R.T. can tell you through diagnostics

author: magic
author_rank: Super Contributor
timestamp: August 31, 2025, 04:52:43 pm
content: Any half-decent USB-SATA bridge supports SAT so "-d sat" is most likely to work on unknown chips.Case in point, recent smartmontools has a drivedb entry to use SAT with this JMicron chip.

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2025, 05:18:50 pm
content: tried with a different usb-sATA adapter, lent to me by a friendCode:lelly ~ # smartctl -d sat -a /dev/sdbsmartctl 5.39.1 2010-01-28 r3054 [powerpc-unknown-linux-gnu] (local build)=== START OF INFORMATION SECTION ===Device Model: Hitachi HTS725050A9A362Serial Number: 100601PCG420VLJ070MCFirmware Version: PC4ACB1EUser Capacity: 500,107,862,016 bytesDevice is: Not in smartctl database [for details use: -P showall]ATA Version is: 8ATA Standard is: ATA-8-ACS revision 6Local Time is: Sun Aug 31 15:14:00 2025 CESTSMART support is: Available - device has SMART capability.SMART support is: Enabled=== START OF READ SMART DATA SECTION ===SMART overall-health self-assessment test result: PASSEDGeneral SMART Values:Offline data collection status: (0x00) Offline data collection activitywas never started.Auto Offline Data Collection: Disabled.Self-test execution status: ( 0) The previous self-test routine completedwithout error or no self-test has everbeen run.Total time to complete Offlinedata collection: ( 645) seconds.Offline data collectioncapabilities: (0x5b) SMART execute Offline immediate.Auto Offline data collection on/off support.Suspend Offline collection upon newcommand.Offline surface scan supported.Self-test supported.No Conveyance Self-test supported.Selective Self-test supported.SMART capabilities: (0x0003) Saves SMART data before enteringpower-saving mode.Supports SMART auto save timer.Error logging capability: (0x01) Error logging supported.General Purpose Logging supported.Short self-test routinerecommended polling time: ( 2) minutes.Extended self-test routinerecommended polling time: ( 131) minutes.SCT capabilities: (0x003d) SCT Status supported.SCT Feature Control supported.SCT Data Table supported.SMART Attributes Data Structure revision number: 16Vendor Specific SMART Attributes with Thresholds:ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE1 Raw_Read_Error_Rate 0x000b 099 099 062 Pre-fail Always - 1966082 Throughput_Performance 0x0005 100 100 040 Pre-fail Offline - 03 Spin_Up_Time 0x0007 130 130 033 Pre-fail Always - 34 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 6845 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 2487 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 08 Seek_Time_Performance 0x0005 100 100 040 Pre-fail Offline - 09 Power_On_Hours 0x0012 025 025 000 Old_age Always - 3293810 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 012 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 406160 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0191 G-Sense_Error_Rate 0x000a 100 100 000 Old_age Always - 21474836480192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 139193 Load_Cycle_Count 0x0012 001 001 000 Old_age Always - 2833649194 Temperature_Celsius 0x0002 189 189 000 Old_age Always - 29 (Lifetime Min/Max 11/59)195 Hardware_ECC_Recovered 0x000a 100 100 000 Old_age Always - 0196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 335197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 9198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0223 Load_Retry_Count 0x000a 100 100 000 Old_age Always - 0254 Free_Fall_Sensor 0x0032 100 100 000 Old_age Always - 12SMART Error Log Version: 1ATA Error Count: 889 (device log contains only the most recent five errors)CR = Command Register FR = Features Register SC = Sector Count Register SN = Sector Number Register CL = Cylinder Low Register CH = Cylinder High Register DH = Device/Head Register DC = Device Command Register ER = Error register ST = Status register Powered_Up_Time is measured from power on, and printed asDDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,SS=sec, and sss=millisec. It "wraps" after 49.710 days.Error 889 occurred at disk power-on lifetime: 32936 hours (1372 days + 8 hours)When the command that caused the error occurred, the device was active or idle.After command completion occurred, registers were:ER ST SC SN CL CH DH-- -- -- -- -- -- --40 51 04 0c d0 fb 40 Error: UNC 4 sectors at LBA = 0x00fbd00c = 16502796Commands leading to the command that caused the error were:CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name-- -- -- -- -- -- -- -- ---------------- --------------------25 00 08 08 d0 fb 40 00 21:41:57.200 READ DMA EXT25 00 08 08 d0 fb 40 00 21:41:53.900 READ DMA EXT25 00 08 00 d0 fb 40 00 21:41:53.900 READ DMA EXT25 00 08 f8 cf fb 40 00 21:41:53.900 READ DMA EXT25 00 08 f0 cf fb 40 00 21:41:53.900 READ DMA EXTError 888 occurred at disk power-on lifetime: 32936 hours (1372 days + 8 hours)When the command that caused the error occurred, the device was active or idle.After command completion occurred, registers were:ER ST SC SN CL CH DH-- -- -- -- -- -- --40 51 04 0c d0 fb 40 Error: UNC 4 sectors at LBA = 0x00fbd00c = 16502796Commands leading to the command that caused the error were:CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name-- -- -- -- -- -- -- -- ---------------- --------------------25 00 08 08 d0 fb 40 00 21:41:53.900 READ DMA EXT25 00 08 00 d0 fb 40 00 21:41:53.900 READ DMA EXT25 00 08 f8 cf fb 40 00 21:41:53.900 READ DMA EXT25 00 08 f0 cf fb 40 00 21:41:53.900 READ DMA EXT25 00 08 e8 cf fb 40 00 21:41:53.900 READ DMA EXTError 887 occurred at disk power-on lifetime: 32936 hours (1372 days + 8 hours)When the command that caused the error occurred, the device was active or idle.After command completion occurred, registers were:ER ST SC SN CL CH DH-- -- -- -- -- -- --40 51 eb 85 d0 fb 40 Error: UNC 235 sectors at LBA = 0x00fbd085 = 16502917Commands leading to the command that caused the error were:CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name-- -- -- -- -- -- -- -- ---------------- --------------------25 00 f0 80 d0 fb 40 00 21:41:49.500 READ DMA EXT25 00 10 70 d0 fb 40 00 21:41:49.500 READ DMA EXT25 00 f0 80 cf fb 40 00 21:41:46.100 READ DMA EXT25 00 10 70 cf fb 40 00 21:41:46.100 READ DMA EXT25 00 f0 80 ce fb 40 00 21:41:46.100 READ DMA EXTError 886 occurred at disk power-on lifetime: 32936 hours (1372 days + 8 hours)When the command that caused the error occurred, the device was active or idle.After command completion occurred, registers were:ER ST SC SN CL CH DH-- -- -- -- -- -- --40 51 64 0c d0 fb 40 Error: UNC 100 sectors at LBA = 0x00fbd00c = 16502796Commands leading to the command that caused the error were:CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name-- -- -- -- -- -- -- -- ---------------- --------------------25 00 f0 80 cf fb 40 00 21:41:46.100 READ DMA EXT25 00 10 70 cf fb 40 00 21:41:46.100 READ DMA EXT25 00 f0 80 ce fb 40 00 21:41:46.100 READ DMA EXT25 00 10 70 ce fb 40 00 21:41:46.100 READ DMA EXT25 00 f0 80 cd fb 40 00 21:41:46.100 READ DMA EXTError 885 occurred at disk power-on lifetime: 32935 hours (1372 days + 7 hours)When the command that caused the error occurred, the device was active or idle.After command completion occurred, registers were:ER ST SC SN CL CH DH-- -- -- -- -- -- --40 51 e8 14 e0 bb 4c Error: UNC 232 sectors at LBA = 0x0cbbe014 = 213639188Commands leading to the command that caused the error were:CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name-- -- -- -- -- -- -- -- ---------------- --------------------25 00 e8 14 e0 bb 40 00 21:14:14.600 READ DMA EXT25 00 f0 0c e0 bb 40 00 21:14:11.100 READ DMA EXT25 00 08 04 e0 bb 40 00 21:14:11.100 READ DMA EXT35 00 08 74 d0 ff 40 00 21:14:11.100 WRITE DMA EXT35 00 08 f4 cf 3b 40 00 21:14:11.100 WRITE DMA EXTSMART Self-test log structure revision number 1No self-tests have been logged. [To run self-tests, use: smartctl -t]SMART Selective self-test log data structure revision number 1SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS1 0 0 Not_testing2 0 0 Not_testing3 0 0 Not_testing4 0 0 Not_testing5 0 0 Not_testingSelective self-test flags (0x0):After scanning selected spans, do NOT read-scan remainder of disk.If Selective self-test is pending on power-up, resume after 0 minute delay.Code:lelly ~ # smartctl -d sat -t long /dev/sdbsmartctl 5.39.1 2010-01-28 r3054 [powerpc-unknown-linux-gnu] (local build)=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===Sending command: "Execute SMART Extended self-test routine immediately in off-line mode".Drive command "Execute SMART Extended self-test routine immediately in off-line mode" successful.Testing has begun.Please wait 131 minutes for test to complete.Test will complete after Sun Aug 31 17:28:41 2025Use smartctl -X to abort test.let's wait ... for test to complete

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 31, 2025, 05:47:37 pm
content: Quote from: DiTBho on August 31, 2025, 12:34:38 pm"Sense Key : 0x3 " and "ASC=0x11" means an unrecoverable read error from the medium occurred. This almost always indicates a physical problem, like a dust mote, or damage to the surfaces within that sector. Usually a head impact damages a run or a set of sectors, though. (If one or more heads are damaged, nothing on the disk is readable.)If they all happened on the same sector (498865340) within the inode table, at best you lost nothing at all, and at worst you lost some entire files.

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2025, 06:03:55 pm
content: Quote from: Nominal Animal on August 31, 2025, 05:47:37 pmI was expecting something like this too, but look above... first it tells me there are errors while I'm copying files, but by trying again, I can copy them and they're not corrupted.I can't even run ls of sbin/ bin/ usr/bin, ... and here too... sometimes it can read some executables correctly (e.g. mount, /sbin/agetty), and they're not corrupted, other times it says it can't read them.no way to read root ... this is gone permanently ...But why this strange, random behavior? Either the diskblock of file/folder is broken, or it's not broken, or it's damaged, or it's not damaged.And above all, how could a disk that has never been hit and has always worked inside a Mac Mini fail like this?

author: Nominal Animal
author_rank: Super Contributor
timestamp: August 31, 2025, 06:22:06 pm
content: Quote from: DiTBho on August 31, 2025, 06:03:55 pmYes, because the read error occurred in the inode table.For those not aware, in Unixy filesystems, an inode is the object which describes the owner, permissions, timestamps, and where the data chunks are, for a filesystem object (file or directory); but not the name. The names are listed in directory objects, which refer to the inodes by array index. Thus, when some of the inode table cannot be read, some filesystem objects become completely unreachable, because those inodes are now inaccessible and lost. You still have their names, and every file you can open and read will be undamaged; its just that some inodes, everything except their name, may be lost. DiTBho may even have gotten lucky, with the inode table (which is usually fixed in size and location when the filesystem is formatted) having only unused inodes in the damaged sector, in which case no harm.If DiTBho got no errors from the filesystem layer, only from the hardware layer, that means that part of the inode table is unused and was only read due to readahead. This would basically be the optimum case: nothing at all lost, and a physical error detected due to readahead for data not currently used in the storage medium at all.Quote from: DiTBho on August 31, 2025, 06:03:55 pmPlain wear, or a single dust mote. It does happen, it is just rarer than the other types of failure.

author: m k
author_rank: Super Contributor
timestamp: August 31, 2025, 07:05:16 pm
content: When originally less than good sector becomes worse enough its read speed lowers.At some point it becomes slower than accepted and generates an error.Next read can be fast enough and is accepted.For example an older stuff may have a regular speed of 1 and 500 is still accepted.When next write happens the disk decides what to do, is it good enough or should it be remapped.

author: amyk
author_rank: Super Contributor
timestamp: August 31, 2025, 08:17:34 pm
content: Code:193 Load_Cycle_Count 0x0012 001 001 000 Old_age Always - 2833649It's been killed by excessive load/unload cycles caused by a power-saving feature. I've read that this wears the parking ramp down, creating particles that find their way to the heads and destroy them.

author: DiTBho
author_rank: Super Contributor
timestamp: August 31, 2025, 09:46:43 pm
content: Quote from: amyk on August 31, 2025, 08:17:34 pmAs also reportedhere, andhere. Very interesting.

author: DiTBho
author_rank: Super Contributor
timestamp: September 02, 2025, 10:25:10 am
content: QuoteCode:smartctl -s apm,off /dev/sdamust be prepared an /etc/init.d/hdd_apm service for this!

author: DiTBho
author_rank: Super Contributor
timestamp: September 02, 2025, 10:38:24 am
content: Umm, I just realized that my home was not backup up and there was an article I was writing, plus a tool I wrote in C to help me with "markdown" stuff ... all gone, d'oh.Is there any way to recover? I cannot access my home directory, "cd /home" reports a lot of IO errors ... umm, should I access the hdd at the driver level (/dev/sda), and hex-search for stuff?any better idea?

author: magic
author_rank: Super Contributor
timestamp: September 02, 2025, 11:24:30 am
content: You could try photorec, it also supports text files.Maybe xfs_repair could find lost inodes if a directory is screwed up, but not sure. I would only run it on an image or a COW layer over the actual disk, no raw disk itself.EXT4 is easier to recover data from.

author: DiTBho
author_rank: Super Contributor
timestamp: September 02, 2025, 12:26:55 pm
content: Quote from: magic on September 02, 2025, 11:24:30 amIn testing, ext4 performed worse than xfs on my Mac mini/x86.Years ago, a bug in a bash script caused "rm -r sub-tree" to be run (instead of just cleaning bak and other junk files, it deleted everything) when resyncing (backing up) a project, thus deleting all the files BEFORE they were copied remotely. No local files, no remote files. Great! A silly bug, and it was very difficult to recover them. At least it wasn't a dying HDD.Yes, I had the previous backup, plus a manual backup, but that's not the point: in my experience, ext3 and ext4 were no easier to recover with ext-tools.

author: DiTBho
author_rank: Super Contributor
timestamp: September 02, 2025, 03:38:34 pm
content: Load_Cycle_Count...So,Load_Cycle_Countis one of the thing you should look at.What do you else monitor?--health reports nothing in my case, as if the HDD had not manifested any physical damage.

author: Nominal Animal
author_rank: Super Contributor
timestamp: September 02, 2025, 07:23:30 pm
content: Statistically (observed in computing clusters), when spinning disks exhibit an increase in the raw reallocated sector count (other than in an isolated step by one), it is likely to die in the next week or so; reliable enough for myself to put it in the to-be-replaced queue. (I also compared it to those after initial self-tests, and after a full-disk format followed by self-tests: any change, and I'd consider the disk a Monday unit, not to be trusted at all.) An increase in the raw raw read error rate can be an even earlier sign, but is much more uncertain.Google used to have a PDF on their statistics on this back when they used spinny rust, but I forget the name and location; it was over a decade ago.

author: amyk
author_rank: Super Contributor
timestamp: September 03, 2025, 02:39:31 am
content: Get a new drive of the same or larger size andddrescuethe failing one to it ASAP if you don't want to try a professional data recovery service ($$$).I suspect you may find that 1/4 of the data is gone, as one of the heads is dead, but the other 3 may still be working... for the time being.If you are feeling really adventurous, and after you have ddrescue'd what you can, you may try doing a head swap.

author: DiTBho
author_rank: Super Contributor
timestamp: September 03, 2025, 11:58:40 am
content: Quote from: magic on September 02, 2025, 11:24:30 amit generated 16GB of truncated files.Not useful in my case.

author: DiTBho
author_rank: Super Contributor
timestamp: September 03, 2025, 12:05:48 pm
content: Quote from: Nominal Animal on September 02, 2025, 07:23:30 pmI am planning to write a custom monitor that queries smartctrl.So, it will also keep an eye on the increase in raw reallocated sector count.Quote from: Nominal Animal on September 02, 2025, 07:23:30 pmJust found it! Very interesting, just what I needed to read!

author: DiTBho
author_rank: Super Contributor
timestamp: September 03, 2025, 12:12:48 pm
content: Quote from: amyk on September 03, 2025, 02:39:31 amalready cloned, just to male a few tests.~12GB gone on a 500GB total space.I lost my Home folder, in particular the article I was writing. Less than 200Kbyte, but full of personal notes.I am trying to recever manually by extracting chunks of data directly from the devname.Quote from: amyk on September 03, 2025, 02:39:31 amHow do you know one head is dead?How do I open the disc without letting tiny specks of dust in that could further damage the platters?

author: DiTBho
author_rank: Super Contributor
timestamp: September 03, 2025, 04:08:50 pm
content: attached docs:disk failures, Google reportmonitoring hard disks with smartmontools, article

author: joeqsmith
author_rank: Super Contributor
timestamp: September 03, 2025, 04:19:45 pm
content: Had a drive fail at work and potentially lost a few weeks of work. I pulled it apart and the Kapton ribbon cable that runs to the head had grown a whisker that had shorted two of the traces together. I used a power supply to clear the fault. It worked long enough for me to pull the data. I may have some photos of it.

author: tooki
author_rank: Super Contributor
timestamp: September 03, 2025, 04:52:17 pm
content: Quote from: DiTBho on August 31, 2025, 06:03:55 pmI used to work in IT, and I would hazard that the vast majority of hard drive failures occur in drives that never experienced any shock or other external influence. They just… die.

author: DiTBho
author_rank: Super Contributor
timestamp: September 03, 2025, 05:39:25 pm
content: Quote from: joeqsmith on September 03, 2025, 04:19:45 pmVery interesting.Did you also monitor the hdd before it failed?How did the failure occur?In my case, I remember it very well. I was connected to the Mac-mini/x86 remotely from my laptop using mobaxterm (it's a Windows ssh client). I had a "c/89" file open on the Mac-mini/x86 with GNU/Nano. I was studying it, but since it mentioned things I didn't know, I opened a browser on my laptop to do a Google search.I basically left the Mac-mini/x86 sitting on that file for ~15 minutes, without letting it do anything else, and I think the disk went idle due to the silly "APM" feauture.Both the Mac-mini/ppc and Mac-mini/x86 use an 2.5" hdd, and 2.5" harddisks are made for laptops, so APM is always on unless you disable it. But I didn't.Then I went back to mabaxterm and opened a new ssh session to the Mac-mini/x86. Up to that point, no errors(1) ... but as soon as I "cd /src/dev/idp" to go and study another project... all sorts of errors immediately appeared.from "sorry, I can see the directory, but I can't access it"to "sorry, I'm getting read IO errors..."And from that point on, things just got worse: { "mount", "cd", "ls", "chroot", "agetty" ... } ... a lot of binaries in { /bin, /sbin, /usr/bin, /usr/sbin } were broken... Basically, the system crashed, and upon reboot, it was no longer able to mount the disk or log in.So, then I mounted the disk on a USB adapter to Mac-mini/ppc, and that's where it is today ...(1) Note that up until now, my home directory was accessible because when you connect with ssh, the server changes the current directory to your home directory... yet, after I used the "cd /src/dev/idp" command to view the projects directory, when everything went haywire and my home directory became completely inaccessible.There are possible explanations. Perhaps the home directory was still in the disk cache, or in the kernel-fs cache, and the disk didn't need to physically access it.

author: magic
author_rank: Super Contributor
timestamp: September 03, 2025, 05:55:15 pm
content: Quote from: DiTBho on September 03, 2025, 12:12:48 pm12GB is quite a lot if that's the total of unreadable sectors. Or is it the total of files lost from filesystem corruption?Quote from: DiTBho on September 03, 2025, 12:12:48 pmWell, if you know what was there, it should be greppable.Though I would suggest using the image, not the broken disk.

author: DiTBho
author_rank: Super Contributor
timestamp: September 03, 2025, 06:03:59 pm
content: Quote from: magic on September 03, 2025, 05:55:15 pmFrom badblocks, so physical blocks.And it tooks 2 days with 1 { 0x00, 0xaa, 0x55, 0xff } pass!It's become terribly slow, the kernel complains about countless reread attempts, which almost always fail. It really feels like it's dying completely.I've already made an image on a 1T disk.

author: joeqsmith
author_rank: Super Contributor
timestamp: September 04, 2025, 07:08:26 am
content: Quote from: DiTBho on September 03, 2025, 05:39:25 pmI checked my home archive (where I did the deed), and could not find the photos. It was a co-workers drive that worked in the layout department. The computer just up and died then would not boot. IT was planning to replace the drive or PC but for what ever reason, the work was not backed up. They were doing a board for me at the time. Pure luck that I was able to locate that short, blow it out without damaging the drive while exposing it to the outside environment, and actually recover the data.

author: BradC
author_rank: Super Contributor
timestamp: September 04, 2025, 07:43:06 am
content: Quote from: tooki on September 03, 2025, 04:52:17 pmA bit like humans, hard disks come in 2 types. Dead and not dead yet. Some die due to sudden trauma and some just wear out.When you think about the operating tolerances in these devices, it's both a miracle they work at all, and can be made in such quantities they are affordable.

author: Nominal Animal
author_rank: Super Contributor
timestamp: September 04, 2025, 08:04:31 am
content: Quote from: tooki on September 03, 2025, 04:52:17 pmYes. As discussed in the Google paper on their internal statistics on lots of spinny rust: they usually fail without any prior indications, even when well treated. Even they found no reliable indicators at all.Some SMART indicators are useful in that even if the drive is still operating without user-visible errors, increasing read error rate and new reallocated sectors does mean the drive has to compensate to stay operational (compared to its previous state). So, if you are willing to err on the side of caution, it can be sufficient reason to replace the drive.The only thing that works somewhat reliably is backups. Distributed and offline backups if you can, but any kind of backup is always better than none.

author: tooki
author_rank: Super Contributor
timestamp: September 05, 2025, 02:00:31 pm
content: Quote from: BradC on September 04, 2025, 07:43:06 amI have said exactly the same thing for years, practically word for word!!

author: BradC
author_rank: Super Contributor
timestamp: September 05, 2025, 03:25:21 pm
content: Quote from: tooki on September 05, 2025, 02:00:31 pmIt's either a case of "great minds think alike", or "fools rarely differ".Frankly 24TB drives are sufficiently close to magic that I start to feel old.

author: DiTBho
author_rank: Super Contributor
timestamp: September 05, 2025, 04:17:22 pm
content: Quote from: Nominal Animal on September 04, 2025, 08:04:31 amThe document lists some indicators that are necessary but not sufficient to predict a failure.For me, this is sufficient: when any of these indicators occur, it is advisable to replace the HDD, assuming a failure will occur in the near future.I will for sure develop a tool for this, also useful when you have to decide if to take or return "refurbished" HDDs bought on { eBay, Amazon, ... }

author: DiTBho
author_rank: Super Contributor
timestamp: September 08, 2025, 10:33:53 am
content: How can a bldc motor survive billion spins?R.I.P.(now the hdd is officially dead)

author: Nominal Animal
author_rank: Super Contributor
timestamp: September 08, 2025, 11:30:35 am
content: (It is 2,838,240,000 rotations per year when spinning 24/7 at 5400 RPM, and many survive several years even in heavy server use...)The tech we have, when one stops to think about it, is pretty darned amazing.

author: m k
author_rank: Super Contributor
timestamp: September 08, 2025, 01:59:01 pm
content: Fluid bearing.