Using SanDisk Max Endurance series as personal archival media ? - Page 1
Forum section: electronics / general

author: BravoV
author_rank: Super Contributor
timestamp: July 03, 2020, 04:18:38 am
content: For decades, I've been storing personal archive, and thru times had been busying transfering them starting from early eras starting at floppy, tape, IOMega ZIP, CD , DVD, BD disc and currently hard drives (redundantly cloned at 2 identical drives).So far for my upmost important stuffs only < 2TB.I'm particularly interested in choosing new media (not for backup, but archival only) at SanDisk MAX Endurance series microSD card.Link ->DatasheetAs it has impressive claim especially for 256GB version that has endurance of 13 years life with warranty of 15 years, also operating temperature at -25°C to 85°C(-13°F to 185°F), and also marketing jargonslike waterproof, shockproof and X-ray-proof.Even half of the endurance claim at 6.5 years, its still more than enough for my need.Any experience on this stuff ?Again, no cloud thingy discussion please.

author: rdl
author_rank: Super Contributor
timestamp: July 04, 2020, 08:17:48 pm
content: Seems pretty expensive to me. It'll take 8 cards to hold your 2TB. Current price looks like $80+, so almost $700. And that's just for one copy. I prefer quadruple. I use synced copies of mirrored HDDs.

author: BravoV
author_rank: Super Contributor
timestamp: July 05, 2020, 04:46:43 am
content: Yeah, its expensive, the idea of small size, no mechanical and the long endurance time is really tempting.

author: David Hess
author_rank: Super Contributor
timestamp: July 05, 2020, 04:39:27 pm
content: At that price, you could have quadruple or more redundant hard drives. Even 100GB M-DISC Blu-ray media would be less expensive although less convenient to use.MicroSD cards are small enough to easily be lost and are difficult to mark for identification.

author: wraper
author_rank: Unknown
timestamp: July 05, 2020, 04:54:37 pm
content: You would be an idiot for using them as archival storage. Endurance time is not a retention time for which there is no spec. They may start losing your data after a year and still will be in spec. And generally retention time is pretty low for SD cards, don't be surprised if your data gets corrupted after a few years. Not to say that endurance in years is a vague marketing wank. Actual TBW spec is what matters, and they are hiding it.

author: David Hess
author_rank: Super Contributor
timestamp: July 05, 2020, 06:04:47 pm
content: Quote from: wraper on July 05, 2020, 04:54:37 pmThat is a good point. I thought they specified retention in this case.There is a industry specification for minimum retention of Flash memory but it is more than an order of magnitude less than 13 years, and I suspect manufacturers often do not meet it anyway.My current "near line backup" solution is full size 2.5" SSDs which I trust in external USB enclosures. They are less expensive than Flash cards, higher performance, and presumably more reliable if only because they can support power loss protection. The retention issue is hopefully covered by periodically powering them, verifying their contents, and refreshing if necessary. In theory, reading the contents should force scrub on read but I have not been able to verify that.Scrub on read would require power loss protection for safety so I do not think Flash cards can support it safely although Swissbit says theirs do.

author: bw2341
author_rank: Regular Contributor
timestamp: July 05, 2020, 06:22:48 pm
content: I took a look at the other Sandisk products and data retention is not specified for any of them.The only quality claim for the Max Endurance line is endurance and it is specified in a non-standard way in terms of hours of 1080p video written. They don't even specify a bit rate for the video stream.https://www.anandtech.com/show/9248/the-truth-about-ssd-data-retentionI found an Anandtech article on data retention. It summarizes the JEDEC standard for publishing endurance ratings for SSDs. I don't think flash card makers are required to follow the standard. Here's a link to the actual JEDEC presentation:https://www.jedec.org/sites/default/files/Alvin_Cox%20%5BCompatibility%20Mode%5D_0.pdfIn brief, consumer class SSDs have to retain the data for one year after exceeding the Total Bytes Written rating. Enterprise class drives only have to retain for three months. Interestingly, warm operation temperatures lead to less cell wear and cold storage temperatures lead to less charge loss.The thing is, lightly used flash memory should have better data retention, so a higher endurance memory card used for archiving should retain the data for longer. I can't see how this could be tested, except by writing once to a card and keeping it for a decade.

author: wraper
author_rank: Unknown
timestamp: July 05, 2020, 06:38:41 pm
content: Quote from: bw2341 on July 05, 2020, 06:22:48 pmFor faster way testing you could write the data and then read it after say 12, 18, 24... months. At some point of time read speed will start dropping, so you'll know there is a trouble coming. Then will come a time when ECC fails to recover some data.

author: bw2341
author_rank: Regular Contributor
timestamp: July 05, 2020, 07:06:16 pm
content: As the flash cell sizes keep shrinking, we're down to a countable number of electrons for each cell. I wonder if a read cycle is enough to meaningfully impact data retention?

author: BravoV
author_rank: Super Contributor
timestamp: July 06, 2020, 04:50:22 am
content: Quote from: wraper on July 05, 2020, 04:54:37 pmYeah, you're right, dunno what was in my mind.Its the retention that I'm looking for, guess redundant harddrives or SSDs still way to go.But this new Max Edurance series still appealing, as my car dash cam keep eating my microSD like crazy, I will buy the 128GB for it once the current one died. I live in hot climate, literally the heat is killing the card and the dash cam battery.

author: Halcyon
author_rank: Unknown
timestamp: July 06, 2020, 11:15:44 am
content: My experience is don't rely on one medium for backups. Particularly for very important data. You should be maintaining as least 3 copies on various devices for the critical stuff.Consumer flash storage (that includes SD cards, USB storage and SSDs) shouldn't be relied upon for that "crucial" backup when everything goes wrong. Unless you are continuously re-writing and verifying its contents, then you might get away with it.Burnt optical media for the most part is garbage for mid to long-term storage. After about 10 years, I would deem anything written to optical discs as "unreliable".The best way is keep multiple copies on different media and keep "refreshing" it. Utilise checksums (or a filesystem that does it for you, like ZFS) and absolutely don't rely on RAID alone to keep your data safe.

author: PKTKS
author_rank: Super Contributor
timestamp: July 06, 2020, 11:24:43 am
content: 2 cents of jamboI will try to chat will all above comments at once.- I deal with the very same problem over 4 decades at least- same stuff used so far: magnetic/flops/CDs/DVDs..- HDs SSDs NVMs ucards..The last ones ( formerly SSDs and ucards ) are not even closeas reliable as magnetic media, Proof of pudim is last week 2 cardsfailed suddenly without warning - No longer "recognized" by OSThey use obscure storage methods so far unproved in mid to long termMagnetics like flops tapes and HDs are better. HDs at least give yousome hint of incoming failures.CDs DVDs are reliable as mid term - randomly guess in long termWhat is left? a bunch of them sparse distributed combinationswhich more or less will come in the due time.My storage TAG grown to the some dozen Terabytes rangewhich is obviously hard to manage.Paul

author: BravoV
author_rank: Super Contributor
timestamp: July 06, 2020, 11:35:20 am
content: Yep, back to HDs.I do maintain two identical, low hours and low power on count identical HD (from SMART readings).If I'm not lazy , once 6 months, I do check & verify, using RAR, so far after >7 years I guess, not even single bit rot experienced.While optical, I do use QuickPAR, but its all transferred to HDs now.Any idea or suggestion similar to QuickPAR but for HD ?Again, this is NOT for backup, but archival purpose only.

author: David Hess
author_rank: Super Contributor
timestamp: July 06, 2020, 11:45:52 am
content: Quote from: bw2341 on July 05, 2020, 07:06:16 pmRead operations have always affected data retention in NAND Flash memory. It is called "read disturb".

author: PKTKS
author_rank: Super Contributor
timestamp: July 06, 2020, 02:58:28 pm
content: Quote from: BravoV on July 06, 2020, 11:35:20 amThis will undoubtedly hit the kind of setup you have.Mine changed at crazy pace over these decades tryingto accommodate the meg to gig and gig to tera byte rangeand also the speed and the CPU power ratings accordingly.To the point I have never seen a thing so damn good for thatas rsync. Small reliable absolute feature reach and can run unattendedalmost every terminal you can think of.I have tried some more exotic things but to cope with a runningsystem scheduled party you definitely will prefer a mix of dedicatedtrunk ports (link aggregation with fault tolerant and load balancing)so you can route traffic with multiple incomes outcomes easilyAnd ditch those compressed type backups which in the longterm tend to waste CPU power and time and traffic with unnecessarilyrepetitive tasks - sometimes done in the daily basis.All of that with a dedicated port trunking setup (VLAN) capableof handling simultaneously traffic from different origins (servers)rsync can handle that fast and very reliable across multiplefile systems. That last detail is really important when hard linkssymlinks and particular pipes and device type files are considered.so far my routine is that abovePort trunk server (link aggregated fault tolerant VLAN) withmultiple dedicated rsync jobs - fully differentialPaul

author: Mechatrommer
author_rank: Super Contributor
timestamp: July 06, 2020, 04:03:33 pm
content: Code:digital archival = maintenance every X year(s), you can get fancy with special program that utilizes CRC, checksum, redundacy storage etc.HDD is a proven tech and has becoming dirt cheap, why bother with unproven (proven actually with much less retention reliability) and more expensive SSD? my recent failure is PATA HDD from 2007, it did make warning few months ago but what the heck, no really critical files inside i can remake them. i still have few PATA that still readable stripped from few devices (PC and old DSO)i made a habit to label the drive by the year i purchased it (not by my fancy names anymore that proved no practical benefit). if you are too afraid, put a note on it "service me by <insert your favorite year>" modern manufacturing i can reliably say 10 years (from experience) to be safer, service or replace it every 5 years and/or maintain it in every year to ensure integrity, ymmv.

author: David Hess
author_rank: Super Contributor
timestamp: July 08, 2020, 07:02:33 pm
content: And while we are on the topic of Flash memory longevity:https://www.theverge.com/2020/6/24/21301676/tesla-nhtsa-touchscreen-failure-investigation-model-s

author: ChunkyPastaSauce
author_rank: Unknown
timestamp: July 10, 2020, 05:17:10 am
content: Quote from: BravoV on July 06, 2020, 11:35:20 amYou could make an image of the drive to archive, then par the image file.Or maybe look into error checking/correcting fs like ZFS. Saves checksum information on blocks, so it can detect corruption and then rebuild from a copy (the copy can be another disk, or on-disk copies managed by zfs)

author: David Hess
author_rank: Super Contributor
timestamp: July 10, 2020, 09:51:00 am
content: Quote from: ChunkyPastaSauce on July 10, 2020, 05:17:10 amThat is what I do now but PAR can be applied without making an image file.

author: BravoV
author_rank: Super Contributor
timestamp: July 10, 2020, 12:01:29 pm
content: Quote from: David Hess on July 10, 2020, 09:51:00 amDavid, which application do you use ?

author: Fred Basset
author_rank: Regular Contributor
timestamp: July 10, 2020, 12:51:54 pm
content: Quote from: David Hess on July 05, 2020, 04:39:27 pmThank you - I was going to suggest M-DISCs, as these are what I use.With 2TB of data though...how often are you going to be reading or writing this data? Only because I am thinking 2 or maybe 3 hard drives with a copy on each should be enough. Mix the formats, so use one that is a normal HDD, one that is meant for 24/7 video for a security system, and one that is filled with Helium, so probably 10TB capacity. All these together would be only around the $700 for one set of these cards. Mathematically the chance of all 3 failing is so fantastically low that I would not be bothered by it. After all, these HDDs are proven technology these days, while these new cards are new and I would be a little uneasy about how reliable they will turn out to be in the long run? Just a thought...

author: David Hess
author_rank: Super Contributor
timestamp: July 10, 2020, 01:58:36 pm
content: Quote from: BravoV on July 10, 2020, 12:01:29 pmI use Multipar for GUI and it supports directories. I use phpar in batch files when I need faster operation.

author: BravoV
author_rank: Super Contributor
timestamp: July 11, 2020, 05:02:33 am
content: Quote from: Fred Basset on July 10, 2020, 12:51:54 pmEven though I trust the media for longetivity, the idea of keeping a not so reliable optical drive made me reluctant to use it.For decades, all my optical drives starting from CD , DVD up to BD have failed on me, they simply not reliable.

author: Rerouter
author_rank: Super Contributor
timestamp: July 11, 2020, 05:32:47 am
content: My own approach is I bought up a box of 8 old 2TB hard drives of different models and manufacturers for $100 total from an auction, stuck them each in there own sealed plastic container with a stupid amount of desiccant packets, and distributed them between familyIts primarily family photos / artwork scans going back to 1896, so they where happy to put a Tupperware container somewhere safe, and with time scanned there stuff in and added it (at about 945GB of photo scans, because I just went 600dpi for no good reason)Each time I meet up with then through the year for a meal or similar, I leave my laptop to sync / verify the files, and run some tests to make sure they are not degrading. so far 2 of them have failed, but it was cheap enough to replace them, and I know they are safe for now,

author: BravoV
author_rank: Super Contributor
timestamp: July 11, 2020, 06:04:41 am
content: Quote from: David Hess on July 10, 2020, 01:58:36 pmThanks, love it.Its now my primary tool for verification.Just a question, at the option menu, in the "Hardware environment" section, I enabled all features, including the "Enable GPU acceleration", and took few test runs at each run at almost 10 mins, it seems like it doesn't utilize the GPU at all, as the GPU utilization indicator stayed flat at 0, fyi I use AMD RX580.Does yours work using the GPU ?

author: Fred Basset
author_rank: Regular Contributor
timestamp: July 11, 2020, 08:53:42 am
content: Quote from: BravoV on July 11, 2020, 05:02:33 amThat is a good point. Their longevity is predicated on drives always being around that can read them. They have not caught on well enough for me to be confident of that. I have everything on HDDs now anyway, the M-DISCs were supposed to be an emergency archive. When the drives become unavailable then I think I will replace them with a different HDD.Libraries now are facing terrible problems with archiving media that drives for are becoming increasingly difficult to find. I can see me facing the same problem one day if I stick with M-DISCs.If you have a decent connection to the internet, another possible solution is to simply upload the 2TB to several different storage companies. Let them worry about compatability, obsolescence, reliability, etc. One might go down, two at the same time is hard to believe, but three or more should be very safe indeed.

author: David Hess
author_rank: Super Contributor
timestamp: July 12, 2020, 02:34:45 am
content: Quote from: BravoV on July 11, 2020, 06:04:41 amAs far as I know, my GPU is not supported. But I would not use GPU acceleration whether supported or not because GPUs do not correct soft errors.

author: ChunkyPastaSauce
author_rank: Unknown
timestamp: July 12, 2020, 05:26:18 am
content: I have both amd and nvidia cards in my computer, multipar gpu acceleration launched on the nvidia but not the amd. And for nvidia only when I had cuda enabled on it. So I think it doesn't have opencl support. No time difference for 4gb file on my machine.

author: BravoV
author_rank: Super Contributor
timestamp: July 12, 2020, 05:32:07 am
content: Quote from: ChunkyPastaSauce on July 12, 2020, 05:26:18 amThanks for sharing this, guess its only for CUDA, I can live with that.

author: rsjsouza
author_rank: Super Contributor
timestamp: July 13, 2020, 02:23:02 am
content: I usually use an old adage that was also repeated by Louis Rossmann: if a piece of data is not stored in three different places, it is considered lost.I can't afford to have complete separation of places (at home, in the cloud and at a bank safe), but I leave copies in three different places - one of them is a four 3TB HDD RAID5.Unfortunately the amount of significant data took a steep turn up (kids were born, then youtube channel) and therefore optical media did not cut anymore, leaving me only with HDD as a feasible solution.That said, I have 25+ years old CDs recorded in 1x, 2x and 4x that have zero error till this day. I also happen to have a SCSI-2 80MB HDD that still works and has no more errors than its error table. Quite amazing, especially due to the doomsday scenarios of the 1990s regarding magnetic and optical media degradation. Streamer tapes were a disaster for us (1990s 120/250MB), the Iomega solutions were quite good (I was lucky) and I lost count of how many flash drives and pendrives I lost. With this, I still keep all that data on HDDs.