Strange hard disk performance issue - Page 1
Forum section: electronics / general

author: Halcyon
author_rank: Unknown
timestamp: January 25, 2023, 12:11:51 am
content: I have a bunch of HGST 2TB 7K4000-series SATA drives (exact model is HUS724020ALA640) which were pulled from either a NetApp or IBM server some time ago (I don't remember which).I've been attempting to securely erase them, but have found the performance is absolutely shocking, like abnormally so. Chopping it up to a potentially dead drive, I tested a few of the others, same thing. They seem to be reading/writing OK but just extraordinarily slowly (like it would take 3+ days just to format 2TB). Obviously something isn't right here.My only hypothesis so far is that the drives have some kind of custom firmware from NetApp/IBM to prevent disk shucking and re-use in other machines and/or to ensure their customers only buy their "special" disks, but this is just pure speculation at this point.This is not something I've ever run into before, particularly on SATA disks (these aren't even SAS). Has anyone got any thoughts? I have about 10 of these drives and would rather not shred them if they are still perfectly good.

author: MrMobodies
author_rank: Super Contributor
timestamp: January 25, 2023, 12:26:08 am
content: Have you tried MHDD (dos) to see if the sector readings are indeed slow?It reminds me a bit of Connor drives that I found very slow.Back in 2005, I brought 4x 36GB 10,000 rpm scsi drives from an Ebay seller.I connect them up, formatted all them and two failed.I told the seller this and refunded me.Now when I say formatted I mean by the controller before joining them on raid.I noticed that the two that failed were HP branded on the sticker whereas the others were Seagate. Exactly the same drive, model number, shape and pcb board no difference in appearance.

author: JeremyC
author_rank: Regular Contributor
timestamp: January 25, 2023, 01:15:53 am
content: Quote from: Halcyon on January 25, 2023, 12:11:51 amI would try connect the drive to a Linux machine and try write zeros.You must be root to do it.Assuming that this drive is/dev/sdb:dd if=/dev/zero of=/dev/sdb bs=10MDetailed information about "dd":https://www.gnu.org/software/coreutils/manual/html_node/dd-invocation.html#dd-invocation

author: edpalmer42
author_rank: Super Contributor
timestamp: January 25, 2023, 02:51:37 am
content: 1. Use a program like CrystalDiskInfo to check the SMART parameters to see if the drives are just worn out. Look for 'uncorrectable sectors' or 'reallocated sectors'.2. When you say 'format' do you mean a Windows format without the quick format option? If so, you could be bumping up against various OS limitations. Either use the previously mentioned dd under Linux or a utility program from one of the drive manufacturers. I always use Western Digital's Data Lifeguard program. They've discontinued it, but it's widely available at various download sites.3. Do you have any security software running that might be interfering with the format process?4. How are you connecting the drive to your system? Are your drives connecting at SATA 2 or 3 speeds? This drive has a sustained throughput of 171 MB/s which translates to ~1.4 Gbps. If your drive is connected through a USB2 or SATA1 connection your performance will be poor, but shouldn't be as poor as you describe. CrystalDiskInfo will tell you what type of connection you have.

author: Halcyon
author_rank: Unknown
timestamp: January 25, 2023, 05:40:36 am
content: SMART checks out. In any case, I'm experiencing the same issue across multiple (identical) drives.The issue occurs regardless whether I connect it via a USB interface or directly on the SATA controller. It also happens across multiple machines.Windows format, or a utility like DBAN or MHDD produce the same slow results when erasing (about 8 MB/sec). Strangely the MHDD "scan" option appears to read the surface at proper speeds.Doing some further investigation as we speak.

author: DrGeoff
author_rank: Frequent Contributor
timestamp: January 25, 2023, 05:42:28 am
content: Try running iometer HDD profiling and diagnostics to see if they are any use.Iometer projecthttp://www.iometer.org

author: edpalmer42
author_rank: Super Contributor
timestamp: January 25, 2023, 06:05:11 am
content: If they're trash anyway, maybe you could try updating the firmware and see what happens.https://www.dell.com/support/home/en-ca/drivers/driversdetails?driverid=xw1yghttp://files.hddguru.com/download/Firmware%20updates/Hitachi/etc.

author: Halcyon
author_rank: Unknown
timestamp: January 25, 2023, 08:06:32 am
content: Quote from: edpalmer42 on January 25, 2023, 06:05:11 amGood suggestion, I tried the Dell firmware, it fails to install as the "system doesn't meet specifications" (even on an actual Dell machine).I'll try some other sources.From what I've been able to tell, these drives are probably vendor-locked. They work just enough for you to recover data from them, but not usable for actual real-world applications as the write speed is limited.

author: David Hess
author_rank: Super Contributor
timestamp: January 25, 2023, 03:08:13 pm
content: I have used the same drives without problems on my ARECA raid controllers, and with MHDD and HDAT2. One or the other of MHDD or HDAT2 usually runs slow, but the ARECA controller has its own throughput tests which show full speed operation.Take a close look at the SMART diagnostics to see if UDMA error are present which would indicate a problem with the SATA interface.If the drives are going bad, then the reallocated sector count should be high, or the available sectors low, or there should be pending sector reallocations.

author: PA0PBZ
author_rank: Super Contributor
timestamp: January 25, 2023, 04:19:20 pm
content: A vendor lock usually works the other way around, it's not to prevent you from using their drives in another system, it's there so you can only use their drives (bios) in their system and not the oem version.

author: AndyBeez
author_rank: Frequent Contributor
timestamp: January 25, 2023, 08:59:26 pm
content: I'm not sure that I'm bringing anything new to the game here, but one thought is is some hidden or hard to kill EFI (or similar) partition info stuck in a graveyard orbit around the spindle. To burn up the junk requires using the Windowsdiskpartutility; with this you can carry out a (highly) destructivecleanof the disc's information. A clean should considered an option of last resort. A cleansed disc is then reinitialised with a fresh MBR and reformatted as NTFS.btw I had a look at the HGST (Western Digital) datasheet for this model and there is nothing to suggest an OEM or vendor locking option - either onboard or BIOS based. Maybe re-flashing the EEPROM might work, as this could be someunpublishedRAID setup.About diskpart:https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/diskparthttps://www.windowscentral.com/how-clean-and-format-storage-drive-using-diskpart-windows-10

author: nightfire
author_rank: Frequent Contributor
timestamp: January 25, 2023, 11:03:49 pm
content: Also a known fact is that Netapp uses 520 bytes sectoring along with special firmware to do so, at least in the classic machines that used FC based drives or SAS.Could also be a contributing factor to the whole hiccup, as they also use their proprietary WAFL filesystem on them.

author: Halcyon
author_rank: Unknown
timestamp: January 26, 2023, 02:23:06 am
content: Quote from: David Hess on January 25, 2023, 03:08:13 pmZero errors according to SMART and surface read tests check out totally fine. I don't think the drives are faulty at all, it's looking more and more like custom firmware designed to be used in certain servers only.Quote from: nightfire on January 25, 2023, 11:03:49 pmI haven't considered sector size. I'll check that out.Quote from: PA0PBZ on January 25, 2023, 04:19:20 pmYes usually, but it can work the other way around. For example, some manufacturers are vendor-locking AMD CPU's so you can only use their CPU's in their machines (and vice versa). There is some vague information out there about vendor-locked hard disk drives as well. Whilst HGST/WD might not mention it in public specs, that doesn't mean they don't provide the mechanism and firmware to vendors to be able to spin their own custom firmware under some kind of agreement that is not known to the public. Similar kind of agreement to what Dave has with Brymen and flashing/programming his custom multimeters.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: January 26, 2023, 05:43:25 am
content: Such shitcanery.Wasn't it Compaq who used to keep the bios on the HDD instead of on the mainboard?I thought we'd all grown up and stopped doing this rotten bullshit.

author: Haenk
author_rank: Super Contributor
timestamp: January 26, 2023, 09:06:15 am
content: As Test-Software, I'd highly recommend "Hard Disk Sentinel".The odd sector size could really be an issue, chances are you are not able to change it. However another controller might be worth a try.Did you check, if these are OEM drives (with no manufacturer warranty)? Maybe check firmware revision as well?

author: AndyBeez
author_rank: Frequent Contributor
timestamp: January 26, 2023, 09:51:41 am
content: Have you taken a forensic look at the SMART feature set using Linuxsmartctl?https://linux.die.net/man/8/smartctl

author: Halcyon
author_rank: Unknown
timestamp: January 27, 2023, 05:01:58 am
content: Everything looks good on SMART. Disk checks out 100% health. Also checked the sector size, definitely 512 KB.The strangest thing though, I just plugged in a disk to test using HDD Sentinel (to the same machines I've been using) and it seems to be writing at normal speed ~170 MB/sec. I then tested the 3 disks that I have been testing this whole time, and same thing, nil issues. Tried again with MHDD's erase command, everything is fine.Now I can't be certain which disk is which now because I wasn't making note of the serial numbers, but I'm completely confused why all of a sudden things are working. I can't explain it. This isn't the first time I've had this issue with these exact drives, I remember when I first got them about a year ago I was seeing similar behavior so I set them aside and didn't do anything with them until now.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: January 27, 2023, 05:10:10 am
content: Bad SATA cable?

author: Halcyon
author_rank: Unknown
timestamp: January 27, 2023, 05:22:04 am
content: Quote from: Ed.Kloonk on January 27, 2023, 05:10:10 amOn multiple machines? Unlikely. Also, I haven't swapped out any cables and on one of them, the drive goes directly into a dock with a USB 3 interface.Part of me is now sitting here thinking, did I just imagine this whole thing?

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: January 27, 2023, 05:42:40 am
content: Quote from: Halcyon on January 27, 2023, 05:22:04 amDrives are funny.A couple of months ago I had to go away for work. I took along a laptop and WD Red choc full of cat videos and a couple of USB SATA converters. The drive was fine before I left. After a 7 hour drive north from here, I livened up the drive and got all kinds of read error messages, so severe I thought that the car trip stuffed the drive (My driving isn't -that- bad).Since I had considered the drive lost, I kept poking at it every night with SMART short test, which failed each time until one day it didn't. And it hasn't thrown a single error since.Only thing I can put it down to is the drive didn't like the change of location.

author: rdl
author_rank: Super Contributor
timestamp: January 27, 2023, 06:09:55 am
content: Yes, they are.I just set up a new encrypted external drive and when copying files to it the speed was all over the place. The average was barely 30 MB/sec. That's just not right. Internal SATA 7200 rpm HDD to external USB 3 SSD should be more like 80 MB/sec I think. I suspect there's something not right with the front panel USB 3 port, but I haven't had a chance to really look into it.Quote from: Ed.Kloonk on January 27, 2023, 05:42:40 am

author: Nominal Animal
author_rank: Super Contributor
timestamp: January 27, 2023, 09:43:49 am
content: It could be the drive advertises a combination of properties – like NCQ, FUA, 1.5G/3G link speed etc. – that results in most current SATA drivers choosing a subset that the drive itself supports poorly. I'd be particularly suspicious of the power management options, because servers (where these kinds of drives were mostly used) typically mostly ignore those, but desktop machines tend to try to implement. Basically, I suspect the actual firmware on the drives is older, and advertises features that weren't commonly used at the time (on servers at least), but now are, but severely degrade the drive performance.In Linux, I'd look at what properties/parameters your SATA chipset driver provides, and check if overriding (downgrading some of) them yields better performance. If you have both ACPI and libata drivers for your chipset or USB adapter, I'd also check if switching between the two yields a difference in performance.

author: Halcyon
author_rank: Unknown
timestamp: January 27, 2023, 09:50:23 am
content: So weirdly, I've now been able to replicate the problem (kind of). I just don't know how (nor do I know how I fix it).A drive which was suddenly "working" and erasing at full speed using MHDD suddenly reverted back to the slow 8 MB/sec writing speed (but reads at ~170 MB/sec) after I disconnected it from my test PC running MHDD and moved it over to wipe on my normal rig which runs Windows and an older version of the Active@ KillDisk application (the new versions suck).It ran slow on KillDisk, then I moved it back over to the MHDD machine, same cables and everything and it's slow once again there too.EDIT: For shits and giggles I re-ran the erase on the "good" disk (without plugging it into a different machine) and it too is back to the slow pace again.So then, without touching anything, rebooted the system, started a full format using Windows 7, everything is good again.Reboot back into MHDD, full speed once again.This makes absolutely zero sense.

author: Ed.Kloonk
author_rank: Super Contributor
timestamp: January 27, 2023, 09:57:25 am
content: It's punishing you for all that Janet Jackson music.

author: Halcyon
author_rank: Unknown
timestamp: January 27, 2023, 09:57:59 am
content: Quote from: Ed.Kloonk on January 27, 2023, 09:57:25 amCheeky bitch.

author: Halcyon
author_rank: Unknown
timestamp: January 27, 2023, 10:14:39 am
content: Repeated this again (without moving, or touching any cables or drives)...Erase under MHDD = SlowRebooted into Windows 7, ran format = FastRebooted into MHDD = Fast

author: edpalmer42
author_rank: Super Contributor
timestamp: January 27, 2023, 05:11:04 pm
content: When you move drives around are you hot-plugging them or do you power everything down first? Maybe the custom firmware in these drives doesn't like hot-plugging. This could even apply to a USB3 connection.Ed

author: Monkeh
author_rank: Super Contributor
timestamp: January 27, 2023, 05:31:35 pm
content: Quote from: Halcyon on January 27, 2023, 10:14:39 amAre you erasing the entire device or partitions?

author: MrMobodies
author_rank: Super Contributor
timestamp: January 27, 2023, 08:57:05 pm
content: Quote from: Halcyon on January 27, 2023, 09:57:59 amJoke: Janet Jackson is hurting their heads so they can't think properly and therefore slow down. I wonder what the heads will have to say about that, calling them cheeky for their high tolerance and choosing wisely not to make mistakes at the expense of speed.Quote from: Halcyon on January 27, 2023, 10:14:39 amHave you noticed or tried taking note to see if the drive parameters have changed when it works fast and when it goes slow like the ones here:I remembered about 7 years ago there was some DMicros video recorders that had an IDE interface and couldn't get many IDE drives new anymore. We tried different sata converters where it didn't pick them the drive up or it just hung on detecting the drive. I brought in a few and this Silicon Image controller (I found them very reliable) worked where it did detect the drive and loaded up but as soon as formatting them it would never finish.Some brand new IDE Hitachi drives were found and still being made but the recorders wern't detecting them. I tested the drives and they were fine. I compared a used Western Digital 500GB that worked and the new Hitachi drives that weren't being detected and noticed a difference in "ATA version". I am not sure myself if that had something to do with it.

author: Halcyon
author_rank: Unknown
timestamp: January 27, 2023, 11:11:46 pm
content: Quote from: edpalmer42 on January 27, 2023, 05:11:04 pmAlways powering down. The USB3 dock I use is a Tableau forensic dock with it's own power switch to power off drives before removing them.Quote from: Monkeh on January 27, 2023, 05:31:35 pmEntire device using MHDD/DBAN/DiskKill, and partition when we're talking about Windows format.Quote from: edavid on January 27, 2023, 05:36:57 pmI've tried with the BIOS functions disabled previously and it didn't seem to make a difference. As I mentioned before, this behaviour has been observed across 3 different computers. I'm 99% certain it has something to do with the drives. I just don't know what. There doesn't seem to be an obvious reason but whatever it is I believe is deliberate (unless a whole batch of drives suddenly became faulty somehow).Quote from: MrMobodies on January 27, 2023, 08:57:05 pmThat was going to be my next task, was to try and view the drive parameters and compare.