Probable cause for Intel SSD failures - Page 1
Forum section: electronics / general

author: cavac
author_rank: Contributor
timestamp: December 28, 2018, 01:45:10 pm
content: We had a few unexpected (early) failures of Intel solid state drives at work. Googling a bit, we seem to be in good company.I had a few minutes and disassembled a failed drive after trying software fixes. I posted my analysis plus pictures on my blog:https://cavac.at/guest/blog/view/94Can't be sure about it, but the heat/thermal pad on the controller chip seems to leak some sticky residue that might or might not be slightly conductive and/or corrosive.And the chip itself has no underfill. Especially on worksloads in strong short bursts, one can imagine quite a bit of thermal cycling that stresses the solder balls.I'm just an I.T. guy who sometimes plays with Arduinos, so i'm not sure my analysis is correct, though. What do you think?

author: wraper
author_rank: Unknown
timestamp: December 28, 2018, 03:44:40 pm
content: QuoteRed herring.

author: wraper
author_rank: Unknown
timestamp: December 28, 2018, 03:46:43 pm
content: About clearing suspicions of being conductive, stick multimeter probes into thermal pad and try measuring resistance.

author: Benta
author_rank: Super Contributor
timestamp: December 28, 2018, 08:24:55 pm
content: Failure mode #1 for SSDs is the IT department, loading so many logging/diagnostic/antivirus etc. crap tools on the PCs that the SSDs simply wear out. As most of them don't have an activity LED like hard drives, it's never noticed.Harddisks survive the SysAdm terror better, but at some point mechanical fatigue steps in.Stop looking at "sticky residue" and look at all the scripts etc. you're loading onto the poor PCs. That'll bring you further.Sorry, but I had to vent. All the crap the IT guys are installing on our machines make them operate at PC-XT speed.

author: wraper
author_rank: Unknown
timestamp: December 28, 2018, 08:44:21 pm
content: Quote from: Benta on December 28, 2018, 08:24:55 pmCannot be further from truth. Unless you run special tools which test SSD endurance, it's very hard to actually exceed specified write cycles of NAND. And what you listed are certainly not among those tools which can make hundreds of terabytes written onto SSD. What often cause SSD failure is that firmware freaks out due to some bug or some other reason and locks SSD. Without actual hardware failure.

author: Mr. Scram
author_rank: Super Contributor
timestamp: December 28, 2018, 08:46:34 pm
content: Quote from: Benta on December 28, 2018, 08:24:55 pmI'm not really buying that story. Empirical data shows you need to write an excessive lot to wear an SSD out. Unless you're running artificial benchmarks to do exactly that, it's very unlikely you'd ever run into issues.

author: Mr. Scram
author_rank: Super Contributor
timestamp: December 28, 2018, 08:49:06 pm
content: Quote from: wraper on December 28, 2018, 08:44:21 pmTests show most drives go between 700 TB and a petabyte. That's really a massive amount. Even if you consider things like write amplification, it's very hard to run into that limit if you're not continuously swapping data into large databases and such.

author: floobydust
author_rank: Super Contributor
timestamp: December 28, 2018, 09:01:25 pm
content: Sandforce SSD drive controllers are the least reliable, total crap. RIP Sandforce.

author: Mr. Scram
author_rank: Super Contributor
timestamp: December 28, 2018, 09:07:54 pm
content: Quote from: blueskull on December 28, 2018, 08:58:05 pmWhere do you get the 5x write amplification and 3 years projected life from? Those numbers aren't anywhere near what the empirical data shows. Unless you're an idiot using consumer grade drives in a main server swapping data to databases continuously or are modelling the same on a desktop, you're not going to get to those kinds of numbers. Considering we were talking about "logging/diagnostic/antivirus"here it's simply not going to happen on that basis.

author: Benta
author_rank: Super Contributor
timestamp: December 28, 2018, 09:11:36 pm
content: Quote from: wraper on December 28, 2018, 08:44:21 pmWho is talking about 100s of Terabytes? Not me!Just to give you a few figures, as SSDs are mostly NAND-based:NOR Flash (which is far more reliable than NAND Flash) guarantees 100,000 erase cycle lifetime over temperature forevery cell.NAND Flash doesn't give any guarantee at all, just a probability of bad sectors appearing anytime.NOR Flash guarantees at least 10 years data retention, some suppliers 20 or more years.NAND: no guarantee, it can fail any time.Constantly writing to specific areas, especially the boot sector, which SysAdms and OS updates love to do will kill an SSD in short time.I've personally had the same experience as the OP with several PCs with SSDs and have reverted to magnetic storage for the same reason.It's a software problem, the SSDs are well specified.

author: thm_w
author_rank: Super Contributor
timestamp: December 28, 2018, 09:12:15 pm
content: There may be more to the drive than just the firmware, perhaps internal calibration/bad NAND storage tables, etc.?Did you completely wipe the drive after updating the firmware, or was that even possible?I bought a refurb SSD a while ago, after a year the thing died (similar symptom, won't detect or respond to commands). Now I'm almost certain all they did to "refurb" the drive was just low level clean wipe the drive, and the same issue occurred again, some kind of NAND corruption or similar, that the controller could not deal with. You would need their software and probably JTAG cable, to allow this level of access.I don't think heat is much of an issue for these drives, but I could be wrong.

author: wraper
author_rank: Unknown
timestamp: December 28, 2018, 09:25:29 pm
content: Quote from: Benta on December 28, 2018, 09:11:36 pmDo you realize all modern SSDs have wear leveling? If you white the same sector multiple times, each time actual data will end up written somewhere else. It's why you won't get one area wear out while other stay not written. As a downside, that adds complexity. Thus prone to firmware bugs or slight data corruption which may cause firmware to lock a whole SSD.

author: tsman
author_rank: Frequent Contributor
timestamp: December 28, 2018, 09:27:08 pm
content: My experience has been that most of the issues are due to buggy firmware or abrupt power loss. The internal flash metadata gets corrupted and the controller can't cope so just goes read only in an attempt to protect the data or just outright crashes. Same thing happens with regular flash media like SD cards. Some SSDs have large banks of capacitors in an attempt to do a more graceful shutdown.If you're lucky then there will be a special utility that sends it manufacturer specific (and undocumented) commands to low level format it to recover. If you're really lucky then there will be a newer firmware that fixes those issues but SSD controllers change so rapidly, you're lucky if they ever release a single new firmware before moving onto the next buggy iteration. If you're unlucky then its just a paperweight and RMA time if its still in warranty.

author: Mr. Scram
author_rank: Super Contributor
timestamp: December 28, 2018, 09:31:48 pm
content: Quote from: wraper on December 28, 2018, 09:25:29 pmFrom the issues we've seen in various manufacturers, it was fairly obvious they were still discovering how to build a reliable SSD. This lead to the theoretically more reliable SSD to being about as reliable as HDDs. We're past that era though, and SSDs seem to be quite reliable nowadays.Wear leveling and TRIM only was an issue in the very first ancient SSDs. I don't think anyone is really using those any more, if only because Windows won't fit on most.

author: Mr. Scram
author_rank: Super Contributor
timestamp: December 28, 2018, 09:33:17 pm
content: Quote from: tsman on December 28, 2018, 09:27:08 pmIt seems the capacitor banks are slowly disappearing from at least consumer SSDs. Crucial historically had these on their SSDs, but even they are now moving away from them. The story is that they now have ways of dealing with it in software, although it may very well be a cost saving measure in an ever more budget concious market.

author: wraper
author_rank: Unknown
timestamp: December 28, 2018, 09:48:51 pm
content: Quote from: blueskull on December 28, 2018, 09:37:08 pmDunno where you got this figure but it's certainly not true. Link please. You would need to do something very specific achieve something like that (unless it's SSD which is barely used for writing large files and all it's use comes to constantly rewriting small files).

author: dbinokc
author_rank: Contributor
timestamp: December 28, 2018, 11:11:10 pm
content: I had a 4 terabyte pci express drive brick itself after two months of use due to a firmware problem.

author: Mr. Scram
author_rank: Super Contributor
timestamp: December 28, 2018, 11:13:53 pm
content: Quote from: dbinokc on December 28, 2018, 11:11:10 pmI assume that was one of the earlier ones, right? Those were fairly experimental.

author: dbinokc
author_rank: Contributor
timestamp: December 29, 2018, 12:08:55 am
content: I am not sure how early it was, but the last thing I expected was to see a drive costing several thousand dollars fail like that.

author: MrMobodies
author_rank: Super Contributor
timestamp: December 31, 2018, 02:10:41 am
content: I had someone's macbook a couple of months ago. It wouldn't bootup. They had photos on there so I got an adapter and took 2 minutes. Some things wouldn't copy across and it hanged for a little while but fortunately the photos were fine. It only showed 400 hours of power on hours and that wasn't very much since it was made in 2011 but then the customer said she hardly ever turned it on and only used it to upload photos on there. I told her to get a couple of 1tb portable usb drives to put the photos on.Now I got a Sony with Three flash drives in Raid 0.I'll check the the type when I take it out and test it.

author: fzabkar
author_rank: Super Contributor
timestamp: April 03, 2019, 05:11:28 am
content: @Cavac, your photos show a Seagate (aka SandForce) flash controller. It may be that Intel supplied their own firmware, but SandForce controllers were plagued with problems, especially in OCZ SSDs.

author: DC1MC
author_rank: Super Contributor
timestamp: April 03, 2019, 06:03:11 am
content: At least the OP disk (ultra-expensive Intel) has a full metal case and some attempt at thermal relief, you should look at my NAS thread and see pictures of the cheap SSD with full plastic case partitioned in isolated compartments and exactly ZERO thermal relief.The poor Crucial BX500 went in thermal overdrive (72C) with just doing dd if=/dev/zero of=/dev/sdc bs=4k with the PCBoutside of the case, hooked on a USB3.0 adapter.Could you imagine this in a hot laptop in hot weather, the poor disk will kill itself immediately.Also during operation, the flash chips went marginally warmer, but the controller got crazy hot!!!So no matter what I hear about writing boot sector or FAT too often, I still think that the controller is the actual weak point, aided by the non-existent to hostile thermal management on cheap disksDC1MC

author: Zucca
author_rank: Unknown
timestamp: April 03, 2019, 08:01:37 am
content: Quote from: cavac on December 28, 2018, 01:45:10 pmAmbient Temperature?

author: Urs42
author_rank: Unknown
timestamp: April 04, 2019, 01:41:09 pm
content: I did destroy two of four SSDs (Raid 10) in a Server . I did switch parts of the Storage in this Server to NVMe SSDs because the 15k SAS Disks i had before where too slow with the massive amount of I/O that the application on this system does. So i would say that it is possible to destroy a SSD in about 1.5 years. It will NOT happen that fast in a normal desktop system however, some firefox versions did a massive amount of I/O even if the browser is only running in the background, i'm not sure if that is fixed today.