[Warning] Beware of QLC SSDs - Page 1
Forum section: electronics / general

author: blueskull
author_rank: Unknown
timestamp: September 19, 2020, 06:57:11 am
content: It should be obvious that the large DRAM cache and necessity to copy-modify-write 4 bits at a time nature makes QLC SSDs vulnerable to power loss, but I gave my wrong trust to Intel.So, last afternoon after work, my Acer Swift 5 again failed to power off due to an ACPI setup fuck up, and the REISUB trick failed to work, so I had to force power it down.As I arrived at home, it wouldn't boot anymore. As usual, I entered the BIOS, tweaked something (Acer BIOS thinks every time you fail to boot there must be a BIOS fuck up, so it resets the BIOS and enables RST and Secure Boot, which Linux doesn't like), and I was greeted not with the normal Ubuntu circle, but initramfs.I ran fsck, and was then able to boot after having recovered tons of files (didn't remember, but I basically had to hold y key to proceed and there should be at least thousands of messages).Then naturally I checked lost+found, and was horrified by its 26MB size, and at that point I knew I was in trouble.I then cated and zcated the folder and extracted ascii contents, found out most are FireFox cache, so I was relaxed, until I found also sporadically but quite some configuration files and Python scripts of some apps.I spent my next hour trying to find what packages got messed up using debsums, and of course it didn't work -- perl was damaged, so the entire dpkg system is botched.I managed to go through lots of hoops to get perl and debconf to reinstall, and at a moment I thought I lost the OS.Finally, I was able to run debsums and compiled a list of messed up packages, just about to be 540+ of them.I spent my net hour waiting for them to download and reinstall, and I was glad the OS is back, and I didn't lose any personal files really as I routinely backup on every Friday afternoon, so I had the most fresh backup to recover from.Should the file system mishap happen one day earlier, I would have been royally fucked.I was able to find out the root cause that made me had to force power off in the first place -- Linux by default spoofs the BIOS as a Windows device since many BIOSes don't support Linux natively.But more and more UEFI BIOSes support Linux and don't like to be spoofed, so adding "acpi_osi=Linux" in kernel command line solves the problem, just I didn't know beforehand.My ThinkPad seems to be okay with the default kernel command line, but not the Acer.However, by the time the kernel asks the ACPI to power off (seen as it stopped responding to REISUB), it should have already unmounted and synced the SSD, so there is no excuse for the SSD to lose data at that point.So moral of the story? Don't be superstitious to a brand. Even Intel has fallacies, and it can bite. If data integrity and reliability is a thing, get an MLC.I probably won't replace a stock TLC drive, but if I see a QLC ever again, it is going to be replaced with a TLC for casual uses or an MLC with power loss protection for critical uses.At year 2020, QLC is still far from being reliable, even with Intel's engineering.

author: Zbig
author_rank: Frequent Contributor
timestamp: September 21, 2020, 08:15:33 am
content: Thank you for sharing this and for the warning regarding QLC SSDs and let me share a bit of my own advice. Stuff breaks (but you know that). Most probably, that wasn’t the last drive (QLC SSD or otherwise) that will crap out on you. Backing up the stuff you create once a week is nowhere near good enough. For the code stuff, I pretty much commit and push to my NAS-hosted git repository every time I add or change anything of any importance. Before I do that, I consider my data volatile and temporary. Larger stuff like media lives on the NAS as a primary storage and only gets synced to workstation if/when needed. The NAS content itself gets backed up to two remote locations every night. Basically, I’m at the point where losing my system disk is merely an inconvenience and doesn’t involve any significant data loss. I couldn’t feel comfortable otherwise.

author: wraper
author_rank: Unknown
timestamp: September 21, 2020, 08:39:34 am
content: I just wonder why anyone would knowingly buy QLC from Intel. Intel or not, it's still crappy QLC. When you can buy proper (not DRAM-less junk) very fast TLC SSD for about the same price, say ADATA sx8200 pro. Not to say sx8200 pro uses Micron TLC NAND which comes from the same place as Intel QLC NAND (IMFT). And better controller from the same vendor (SMI) as Intel 660p. So the only potential advantage would be a miraculous firmware made by intel, but I doubt it can compensate for dubious hardware.

author: David Hess
author_rank: Super Contributor
timestamp: September 21, 2020, 08:46:27 am
content: MLC (2-bit) and TLC (3-bit) SSDs suffer from the same problem and the solution is the same; sufficient backup power must be provided to complete the current write operation. Technically SLC (1-bit) SSDs also have this problem but in a different way.This is different than being able to complete pending writes which requires even more power, and even fewer SSDs support. This is common in enterprise grade SSDs.The only SSDs I have purchased for myself are ones which specifically include power loss protection.Quote from: wraper on September 21, 2020, 08:39:34 amQLC drives cost less. TLC drives cost less than MLC drives.I am actually somewhat alarmed at how quickly I am burning through the write endurance of the 3D-TLC SSDs that I have. I do not know what other people are doing with theirs, but mine are protected to only last 2.5 years despite having a write endurance 400 times their capacity. MLC drives I have considered have 10 times that write endurance but cost three times the price. The one QLC drive I have considered has half the write endurance.

author: wraper
author_rank: Unknown
timestamp: September 21, 2020, 09:00:38 am
content: Quote from: David Hess on September 21, 2020, 08:46:27 amIn general. When you buy intel 660p you pay TLC price.QuoteQLC writes and reads slower. Needs more fiddling with (virtual) SLC cache and DRAM. Which of course is not good for power loss situations.

author: David Hess
author_rank: Super Contributor
timestamp: September 21, 2020, 09:11:25 am
content: Quote from: wraper on September 21, 2020, 09:00:38 amQLC *is* slower so more backup power will be needed to guarantee the completion of a write or erase operation. Usually this means including a bank of capacitors in the design.

author: indeterminatus
author_rank: Contributor
timestamp: September 21, 2020, 11:31:20 am
content: Quote from: Zbig on September 21, 2020, 08:15:33 amStressing this point as well. Every storage medium is bound to fail at you at some point. A good backup strategy is a necessity. (And no, RAID is not a backup).

author: drussell
author_rank: Super Contributor
timestamp: September 21, 2020, 12:38:47 pm
content: Quote from: blueskull on September 19, 2020, 06:57:11 amWait a minute, do these drives not automatically actually write to flash to flush any write cache after some timeout period? If it was sitting there for at least a few seconds or more after having any write commands sent to it before you killed the power, surely it would have already actually written out any pending writes that were already actually in the drive.Wouldn't the problem actually be that your system crashed/hung while it had a bunch of pending writes in computer ram that had not yet been synced to the drive?Not that I'm saying QLC drives are any good...

author: bingo600
author_rank: Super Contributor
timestamp: September 22, 2020, 06:48:43 pm
content: Quote from: indeterminatus on September 21, 2020, 11:31:20 amWin10 might speed it up, if you have a SSDhttps://www.windowslatest.com/2020/08/15/windows-10-2004-disk-optimizer-reporting-bug/If you're running 2004 , you might want to get this updatehttps://borncity.com/win/2020/08/28/windows-10-version-2004-fixt-update-kb4571744-den-defrag-bug//Bingo - Who's on linux (Mint & Debian)

author: Marco
author_rank: Super Contributor
timestamp: September 22, 2020, 08:49:49 pm
content: I was looking for cheap SSDs with power loss protection guarantee a while back, the Kingston DC1000B looked nice ... didn't get around to building anything with it yet, might be better drives around now.

author: wraper
author_rank: Unknown
timestamp: September 22, 2020, 09:11:14 pm
content: Quote from: Marco on September 22, 2020, 08:49:49 pmIt's two times more expensive than good TLC SSD and has poor performance.

author: Marco
author_rank: Super Contributor
timestamp: September 22, 2020, 10:33:00 pm
content: Is anything without power loss protection good? I'd argue no.

author: drussell
author_rank: Super Contributor
timestamp: September 22, 2020, 11:21:24 pm
content: I've never seen an SSD without at least an attempt at power-loss protection.I don't think that was actually the OP's problem, but, ok....

author: David Hess
author_rank: Super Contributor
timestamp: September 22, 2020, 11:53:51 pm
content: Quote from: drussell on September 21, 2020, 12:38:47 pmThey do complete writes in a timely manner, but there are two problems:1. Data still in the write buffer is lost. This applies to any storage medium as well as data being buffered by the system. Properly designed file systems can handle this with loss of only the data which was in transit.2. An incomplete write operation not only loses the data being written, but can destroy other data in the flash chips and if the flash translation data is damaged, that will render the drive unrecoverable and likely inoperable. The mechanism is obvious for flash which stores multiple bits per cell but it also applies to SLC flash.Quote from: Marco on September 22, 2020, 08:49:49 pmI concluded that if the drive specifications do not list power loss protection as a feature, then they lack it. Kingston makes drives which allegedly include power loss protection like the DC500R and DC500M, but I am not convinced that their consumer level drives like their A400 and KC600 have it; their marketing is deceptive on the subject.Quote from: Marco on September 22, 2020, 10:33:00 pmI completely agree, and that rules out every M.2 drive that I have studied although I have seen photographs of M.2 drives which included it. There are several low cost 2.5" SATA drives which explicitly include power loss protection.Quote from: drussell on September 22, 2020, 11:21:24 pmMany do not, and all M.2 drives that I have looked into so far.

author: wraper
author_rank: Unknown
timestamp: September 23, 2020, 01:25:55 am
content: Quote from: blueskull on September 23, 2020, 01:21:34 amOptane is not even FLASH based. So SLC/MLC or whatever is not even relevant to it.

author: wraper
author_rank: Unknown
timestamp: September 23, 2020, 02:19:10 am
content: Quote from: blueskull on September 23, 2020, 01:57:35 amAgain, pretty much irrelevant point. I would understand if you mentioned that 3D XPoint has much higher random write/read speed, does not have erase-before-write requirement, no need to shuffle data between SLC cache and main TLC/QLC storage, etc. But for some strange reason you compared levels in cell of such completely different technologies, even without mentioning any context. When actually it's as such a minor argument among huge differences in technologies, and they are not even directly comparable. Also data corruption in SSD can happen not because of multiple levels in cell as such but because of incomplete write of some important data. And with more cell levels, write is slower, cache is more actively used, more manipulations with data in background, etc...QuoteFYI I made my point exactly because I know what optane is.

author: BravoV
author_rank: Super Contributor
timestamp: September 23, 2020, 06:37:29 am
content: For power loss protection regarding SSD, does UPS count ?All my desktops and NAS are UPS protected, and once in a while, about every 6 months, I did power outage test "diligently", and all passed and were shut down properly. All my UPSes capable of minimum up time for > 10 minutes.For home & personal use, all those power failure protection SSDs are just not very appealing, at least to me, say ccompared to "cheaper" mainstream desktop performance based SSDs, and not worth the trouble and cost, imo.

author: David Hess
author_rank: Super Contributor
timestamp: September 23, 2020, 08:21:22 pm
content: Quote from: BravoV on September 23, 2020, 06:37:29 amA UPS helps but is not a comprehensive solution.One of the applications where a UPS does not help is removable drives where hot removal is a feature. USB Flash sticks naturally lack power loss protection and also lack other features which makes SSDs more reliable. I have had enough problems with them that I now use a small SSD in place of a Flash stick.QuoteI don't know how much cheaper you were thinking but there are several "cheap" 2.5 inch SATA SSDs which include power loss protection including:Crucial BX500 and MX500Intel S4510And, um, well, that is as far as I have gotten. There are also some candidates from Transcend and Kingston.

author: Marco
author_rank: Super Contributor
timestamp: September 01, 2021, 08:01:25 pm
content: Quote from: David Hess on September 22, 2020, 11:53:51 pmBit of a necro, but I expanded my laptop with a M.2 PM9A3 (after promising the shop I wouldn't claim consumer warranty, Samsung doesn't want to sell these things to consumers). Fitting a PM9A3 is going to be a bit of a botch job on most laptops because of its length though.Got reminded of this thread after my laptop decided to get crashy due to heat from a game and I suspect I screwed the old SSD up by pulling the battery when I shouldn't have. I had grub on the old one as main boot and a complete UEFI boot partition on the PM9A3 (grub can simply boot to that). Just had to pull the old SSD and it worked again.Laptops seem to me the best use case for PLP. You can get tricky with desktops, for instance by lazy backups of snapshots to rotating storage, but not so much on laptops.

author: Doctorandus_P
author_rank: Super Contributor
timestamp: September 03, 2021, 12:12:40 pm
content: Quote from: blueskull on September 19, 2020, 06:57:11 amThere, I corrected that for you.

author: Marco
author_rank: Super Contributor
timestamp: September 03, 2021, 03:27:31 pm
content: That fails to solve the reliability issue.

author: David Hess
author_rank: Super Contributor
timestamp: September 03, 2021, 03:44:09 pm
content: The best solution is to get a drive which includes power loss protection of at least stored data, which includes protection for metadata like the flash translation table. MLC, TLC, and QLC have an obvious failure mode during writes but SLC is not immune to data corruption upon power loss either during a write operation.Even if all writes have been completed, power might be lost while the drive is performing a scrubbing operation. I would assume drives are smart enough not to do this after some sort of shutdown command is issued but what if it is never received?

author: mrflibble
author_rank: Super Contributor
timestamp: September 04, 2021, 07:41:32 am
content: Quote from: David Hess on September 03, 2021, 03:44:09 pmI would assume drives are smart enough to be able to resume operations reliably after such an event.

author: David Hess
author_rank: Super Contributor
timestamp: September 04, 2021, 12:29:24 pm
content: Quote from: mrflibble on September 04, 2021, 07:41:32 amThe problem is that if the scrub operation results in a write, and the power fails during the write, then data may be corrupted. So either the drive needs to have power loss protection or scrubbing has to be disabled before power is removed.

author: mrflibble
author_rank: Super Contributor
timestamp: September 05, 2021, 04:00:19 am
content: Quote from: David Hess on September 04, 2021, 12:29:24 pmOr the drive must be able to recover from a failed scrub operation.

author: David Hess
author_rank: Super Contributor
timestamp: September 05, 2021, 08:29:02 am
content: Quote from: mrflibble on September 05, 2021, 04:00:19 amThe drive needs to be able to recover from a failed write which could be triggered by a scrub operation, which short of a RAID type of organization will be impossible, and maybe even that will not be enough.An interrupted write to a multilevel storage cell destroys any previous data but there is another failure mode which also applies to SLC. The state machine which controls the write operation can run amok if the supply voltage drops out of tolerance, and attempt to write to completely unrelated pages. This exact behavior was observed in drive tests. If those unrelated pages happen to be metadata like the Flash translation tables, then the drive will likely become inoperable.This is completely separate from protecting data in transit which is stored in buffers. If only data in transit was at risk, then interrupted write operations would be no different than those on hard drives and modern filesystems handle this just fine by updating metadata as a separate operation.I do not know why the state machine which controls destructive operations cannot be made proof against power loss however so far all drives with power loss protection rely on sufficient stored energy to complete any write, and presumably erase, operation. For a while SandForce advertised controllers that did not require a capacitor bank for power loss protection but testing showed that they still destructively failed.

author: mrflibble
author_rank: Super Contributor
timestamp: September 05, 2021, 08:39:40 am
content: Quote from: David Hess on September 05, 2021, 08:29:02 amOkay, that would indeed be bad. Do you have a link to those drive tests?

author: David Hess
author_rank: Super Contributor
timestamp: September 05, 2021, 05:42:21 pm
content: Quote from: mrflibble on September 05, 2021, 08:39:40 amNot anymore - they were years ago and nobody seems to like testing drives in this way anymore so there are no newer reports. They used an external circuit to cut power to the drive under control of the Raspberry Pi which was running the test.A search turns up this report but I was thinking of a different one:https://hardware.slashdot.org/story/13/12/27/208249/power-loss-protected-ssds-tested-only-intel-s3500-passes

author: mrflibble
author_rank: Super Contributor
timestamp: September 08, 2021, 08:50:26 pm
content: Quote from: David Hess on September 05, 2021, 05:42:21 pmI suspect there is a correlation between the amount of bricked SSDs in the wild, and the urge to do power failure tests. That's usually how these things come to be. Large amount of consumer reports of problems with drive XYZ, which motivates someone to dig a little deeper.And the reverse of that could very well be what we have now. No epidemic of bricked SSDs, so no power fail tests of the sort you mention.

author: amyk
author_rank: Super Contributor
timestamp: September 13, 2021, 12:53:15 am
content: I suspect "industrial" SSDs that use real SLC flash and are basically designed for embedded systems which don't have any real controlled power-off sequence (e.g. a DOS-based PC where you can just wait for the prompt to appear and turn off the power switch right away) will still pass those tests.Not going to be the fastest nor biggest, and definitely not cheap, but I don't think hundreds of MB/s are really necessary in such applications either, and the ultra-low "seek" time compared to a HDD will already give a good speedup.