Quick vs long format - Page 1
Forum section: electronics / general

author: Rick Law
author_rank: Super Contributor
timestamp: January 02, 2020, 05:48:03 am
content: Happy New Year!I am doing my year end backup, so I was doing things the same way I have been doing for years. Surface scan and formatting new drives to store useful backups. On suspected drives (this year, I have one that all files reads ok but chkdsk doesn't finish), I off-load and surface scan, reformat and then copy back. I know that probably was caused by ultra long filenames. It is a repeatable error with XP era machines. But I surface test and reformat (long instead of quick format) to be sure.Dumb mindless work doing backup is. But this time a thought hit me: With all that intelligence in the drive firmware, it re-allocates bad sector all by itself... Doesn't that make long format and surface scan a waste of time?I sure like to hear your experience/opinion.

author: radar_macgyver
author_rank: Frequent Contributor
timestamp: January 02, 2020, 06:13:52 am
content: The sector reallocation built in to the drive firmware can only work if the drive is asked to read the data in question, so the act of reading data is probably more beneficial than scanning it. Modern filesystems (eg: zfs, btrfs) have a 'scrub' operation, which causes every used sector to be read at least once. If the firmware has to reallocate data, it will do so silently. If it can't, then any redundancy built into the higher level file system is used to re-create the data, and write it back (drive firmware will write it to a different physical sector).

author: Jeroen3
author_rank: Super Contributor
timestamp: January 02, 2020, 07:08:52 am
content: When to use full format:- You are formatting a (software) RAID volume that must be block synchronized.- For cold storage. you are making it your future self easier to read data after a disk crash. All non-relevant data will be erased instead of random.QuoteMost of the time, yes. if you need it because yourdrivethrew an error, the disk is EOL.Best insight into disk health:https://www.hdsentinel.com/

author: David Hess
author_rank: Super Contributor
timestamp: January 02, 2020, 02:10:10 pm
content: Doing a surface scan of the drive by *writing* to every sector will force reallocation on write for bad sectors. Reading the drive will leave bad sectors in place and increase the pending reallocation counter. (1)So my preference now is to write to the entire drive to force reallocation and then check the SMART parameters. This year I found three drives which were in the process of failing.(1) This means that if you have a degraded RAID which keeps dropping a drive and becoming unavailable because of bad sectors, rewriting the bad drive can restore proper operation with bad sector data in only the damaged files unless metadata is affected. I recovered a 4 x 3TB drive RAID5 last year using this method. The lesson learned was *always* do a sector scan before swapping a RAID drive.

author: Jeroen3
author_rank: Super Contributor
timestamp: January 02, 2020, 02:53:10 pm
content: You can also ask the drive firmware to do a surface scan in the background.Extended Self TestI have programmed my server to this once every few months. It takeshoursthough.

author: BravoV
author_rank: Super Contributor
timestamp: January 02, 2020, 04:14:41 pm
content: If you decided to do long format, forget all other 3rd party tools, use the one made by the HD manufacturer.For example Seagate has their called Seatools, WDC also has similar one, as most have the long format process (sometimes they called it full complete diagnostic) that performs whole drive diagnostic starting platter like from sector by sector verification, and up to spindle up and down test and etc.Imo, they know their own stuff much better than other parties.

author: David Hess
author_rank: Super Contributor
timestamp: January 03, 2020, 01:40:12 am
content: I usually find the manufacturer's utilities useless. The long extended test does not do anything more than basic tests and read the entire surface.If you use a wiping utility or a read and then write, then at least sectors which are pending reallocation will be processed. Reading the SMART registers can be very informative as far as drive condition.

author: BravoV
author_rank: Super Contributor
timestamp: January 03, 2020, 05:25:35 am
content: Maybe we're using different tool, in the past I did use SeaTools from Seagate, thru the bootable version as its more powerful than thru Windows version, it managed to scan bad sectors and gave thorough report that Windows OS missed in NTFS full format with full write + verification enabled.https://www.seagate.com/as/en/manuals/software/seatools-bootable/help-topic-bad-sector-found/Quoting"If you give permission to overwrite a bad sector SeaTools will attempt to write a pattern of zeros to that sector. Usually, this action will assist the disc drive firmware in managing the problem by retiring the problem LBA and activating a spare in its place."

author: golden_labels
author_rank: Super Contributor
timestamp: January 03, 2020, 11:32:26 am
content: NOTEEverything here assumes that we are talking about spinning rust, not SSD! While most of this post is generally valid for SSD, in some information may need adjustement.Quote from: Rick Law on January 02, 2020, 05:48:03 amThis is a very good thought. Indeed yes: nowadays firmware is reallocating damaged sectors. If it would occur that any of software scans detects bad sectors, it means that the drive is so horribly damaged that firmware has run out of sectors to reassign (normally each drive has a bunch of those reserved for this purpose, not accessible from the computer’s side).You can see the information about reallocated sectors in your disk’s SMART data. And this is one of the very few SMART values that typically are meaningful in their raw form, indicating the actual number of reallocated sectors across any vendor I’ve heard of. You should observe that value, because if it grows from 0 (or the normalized value falls), it is a clear sign that you should start seeking a new drive and treat this one as soon becoming unreliable.Under Linux you can check SMART data usingsmartctlfromsmartmontools(available as a package in many distros):Code:$ sudo smartctl -a /dev/disk/by-id/ata-WDC_WD1002FAEX-00Y9A0_WD-WCAW34208404smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.3.11-arch1-1] (local build)Copyright (C) 2002-19, Bruce Allen, Christian Franke, [url=http://www.smartmontools.org]www.smartmontools.org[… snip …]SMART Attributes Data Structure revision number: 16Vendor Specific SMART Attributes with Thresholds:ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 53 Spin_Up_Time 0x0027 231 169 021 Pre-fail Always - 14254 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 3065 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 07 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 09 Power_On_Hours 0x0032 015 015 000 Old_age Always - 6242010 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 011 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 012 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 303192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 231193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 76194 Temperature_Celsius 0x0022 101 095 000 Old_age Always - 46196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0[… snip …]Writing and then reading the whole drive, as suggested by earlier posters, will invoke reallocation. But this is a lengthy process, not really suitable for everyday use by most people. A 4TB drive at 200MB/s will take 5.5h to write (only write!). That is for the upper end what you may expect from a 7200RPM drive — for most drives prepare for spending one day per drive, even if they’re smaller. And the information you gain from it is “this drive is unsuitable for further use” if the reallocated sector count grows above 0 during the operation. If it doesn’t, you gain no information at all about its future reliability.Someone has suggested full self-test and that is also a good option, but of course equally time consuming and giving the same type of information. The advantage is that there is a non-zero chance that the vendor has added some extras to the test, ones you can’t perform yourself from the software or that require insider knowledge. While you can perform those from CLI tools (likesmartctlunder Linux), interactive tools may be a better choice here, because they will give you indication of the test progress automatically and also handle cancelling etc. by themselves.Unless you are really inclined to spend your time on testing your drives for backup purposes, I would suggest performing only theshortSMART test to determine if the drive’s circuitry, mechanics and calibration are still OK. You will never have a 100% certain backups, so just “stop worrying and learn to love the bomb”. It is rarely the case that data of size requiring a HDD is worth that much protection. There are exceptions, like data the law obligates you to keep or which, upon being lost, would cause significant financial/image damages to the company. But most likely those are things like movies downloaded from the internet or some other hoarding case, or photos you will probably never look at again. The time needed to perform tests can be spent much better with friends or family.If you want to spend your time on that, I would rather suggest doing checksums of files on the original medium and then, after copying them, on the backup medium. Because that is actually delivering you some information.The truly important data, e.g. private keys or password manager’s database, are no more than tens of megabytes. You can just buy a bunch of cheap USB sticks and store them there regularly, preferably spreading the media physcially across multiple places — in case of some disaster or a theft. Also develop the habit of checking your backups for integrity and have a fuckup policies in place. It is much more important than just blindly copying data.

author: Jeroen3
author_rank: Super Contributor
timestamp: January 03, 2020, 01:00:33 pm
content: For SSD's a full format is meaningless. There is no direct mapping to a block you can use and a place on the chip.Either perform the secure erase pre-formatting (complicated, requires a disk power cycle, special commands, not available in Windows)Instead, in Windows: use a quick format and then optimize the drive (former defragment).In Linux:fstrim

author: SiliconWizard
author_rank: Super Contributor
timestamp: January 03, 2020, 02:36:05 pm
content: Quote from: Jeroen3 on January 03, 2020, 01:00:33 pmYup. I'd even venture that it's also meaningless on many recent HDDs as well, as their embedded controllers are a lot more advanced and probably handle physical formats as they see fit.

author: David Hess
author_rank: Super Contributor
timestamp: January 03, 2020, 02:53:19 pm
content: Quote from: SiliconWizard on January 03, 2020, 02:36:05 pmThe physical layout, rotational latency, and seek time mean that minimizing access time requires the logical and physical layout to correspond except for remapped sectors. It is actually easy to find which sectors are remapped by measuring access time to them.Writing every sector of an SSD is not useless if the SSD is suffering from retention failure which can happen now in months in some cases despite their lying specifications. The SSD has no way to know which sectors are in use or not (TRIM is for performance and cannot be relied on) so reads from bad sectors even though not in use can cause problems with some systems.In theory a full read can force scrub on read for damaged but correctable sectors but it is not clear which drives support this. They often lie.

author: mariush
author_rank: Super Contributor
timestamp: January 03, 2020, 05:13:56 pm
content: I found Western Digital Data Life Diagnostic tool to be enough for detecting weak/bad sectors on mechanical drives and mark them accordingly.LINK :https://support.wdc.com/downloads.aspx?p=3&lang=enUse the ERASE option, which overwrites each byte on the drive, after a restart you'll see the SMART data updated with the number of sectors reallocated and all that.The software works with any drive, not just WD drives.Quick Format just erases the metadata information (the records that hold folders, file names, sizes, last modified and last access times etc) leaving anything else on the drive untouched.Full format, will write and read (testing it was written correctly).SSDs don't store the data sequentially like mechanical drives, in order to extend the life of the flash memory cells which have limited number of erases so the ssd controller has to keep track of which pages of flash are written more often, so the controller will put data in random locations (to keep the wear levels even over the drive) and remember which flash memory corresponds to which actual sector requested by the OS.The SSD also uses some hidden portion of Flash as spare memory and sometimes as write cache and when some pages of flash are too worn out, the controller will start to replace bits of "visible" flash memory with portions of that hidden memory that's less worn out.Because of this even a full format will not erase all data, one could desolder the chips and put them in a test gig or a custom made SSD and retrieve all data stored in Flash and this way you may recover data from outside the portion that's normally visible to the OS.

author: Rick Law
author_rank: Super Contributor
timestamp: January 06, 2020, 01:51:58 am
content: Hey guys, thanks! Lots of useful information here and lots of fruit for thought here.I went radio silent because I shot myself in the foot and blew two drives, so I had a lot more to do than I had expected. I actually connected the molex in reverse. It was firm enough to feel it was right, and I was dumb enough (lazy) to get out from under the desk to get my flashlight to check. So poof, two back-up discs went kaput. Damn, yellow wire to red wire should be easy to spot even in the dark... I didn't think the molex can fit in reverse. It was plug in not all the way but enough for electrical contact, and held firm enough that I thought it was right.Man, I have been connecting molex connectors for 20+ years... First time I did that.Lesson re-learned - check and double check before flipping the ON switch.

author: SiliconWizard
author_rank: Super Contributor
timestamp: January 06, 2020, 04:31:20 pm
content: Quote from: blueskull on January 06, 2020, 01:57:12 amOuch.It unfortunaly shows complete ignorance though, as even for someone that would think M2 could interface with SATA (some actually are SATA I think, as M2 exposes both SATA or PCIe) but obviously still require a proper M2 connection on the motherboard), they should know that a SATA power connector only conveys power supplies and not the data...

author: Jeroen3
author_rank: Super Contributor
timestamp: January 07, 2020, 07:14:36 am
content: If you handle a lot of bare drives. Invest in a docking station or hot-swap bay.Or just a tool-less enclosure. Orico (aliexpress) makes nice ones.

author: Halcyon
author_rank: Unknown
timestamp: January 07, 2020, 10:50:21 am
content: As of Windows 7 (if I recall correctly) a full format zeroes out the disk. Previously it didn't do this.

author: Kilrah
author_rank: Unknown
timestamp: January 07, 2020, 11:40:02 am
content: Welp, if those drives still had Molexes I guess it was time to replace them anywayFor my data storage I have a 2-drive RAID0 set in my desktop PC, and 2 external dual drive RAID-supporting boxes with the same config as backups. I sync the internal to one of the externals every couple of days while the other is stored offsite, both rotated every couple of weeks or so.2-3 times a year I run FreeFileSync in "compare file contents" mode instead of usual "scan for changes" mode in order to force a full read of both main, which would catch failing sectors and potential file corruption. Also check SMART status with CrystalDiskInfo at the same time. If there's any SMART caution the drive gets replaced, from experience when that happens it only takes a few weeks/months until they fail completely.

author: wraper
author_rank: Unknown
timestamp: January 07, 2020, 12:16:35 pm
content: Quote from: Rick Law on January 02, 2020, 05:48:03 amIf you are lucky. Often HDDs refuse to reallocate erratic sectors even if you forcefully try to do them so say by MHDD or Victoria. What I can say for sure, if more than 2-3 bad sectors do appear, and especially if they keep appearing, toss that drive into waste bin before it's too late.

author: wraper
author_rank: Unknown
timestamp: January 07, 2020, 12:33:15 pm
content: Quote from: David Hess on January 03, 2020, 02:53:19 pmWriting every sector is counterproductive. What you will achieve is performance degradation because of no free space left and everything then will need to be erased before writing. Also I don't see how it solves any retention problem. If drive has retention problems, only rewriting actual data has some use. Modern SSDs refresh/move data by themselves anyway. It's a part of wear leveling process.QuoteThey may not know what's abandoned, TRIM helps with that. But they know for sure what's certainly not in use.

author: David Hess
author_rank: Super Contributor
timestamp: January 07, 2020, 03:58:45 pm
content: Quote from: wraper on January 07, 2020, 12:33:15 pmThat is the normal case and further, TRIM is a crutch for poorly designed drives.QuoteSure rewriting the data will work but if the drive is already written and suffering from retention problems, then reading the bad data is useless unless you are trying to recover it. Modern SSDs *might* do idle time scrubbing but that is no help if left unpowered and is no help if the manufacturer lied about it. In the later case, the same applies to scrub on read.Idle time scrubbing and scrub on read present a conundrum to the manufacturer because supporting them means the drive must also support power loss protection. Otherwise power loss during a scrubbing operation can potentially wreck the drive. Tests in the past have shown that very few drives, including those which say they do, correctly support power loss protection.QuoteAny drive which has been in use for a while will have no unused sectors beyond what TRIM indicates. Further, I have received new SSDs with bad sectors due to retention failure (Crucial) which should only happen with allocated sectors. That caused some interesting file system problems until I got paranoid enough to investigate.

author: Rick Law
author_rank: Super Contributor
timestamp: January 08, 2020, 03:16:50 am
content: Quote from: Jeroen3 on January 07, 2020, 07:14:36 amI actually have a few "hot-swap" bays at hand. Initially, I set up my backup system as a dedicated machine with hot-swap bays. After a few years and with the limited space under the desk, the dedicated system degenerated into just "ah... Just once a year... I can plug the darn drive into my system as needed." The space was gradually filled - taken up by electronic parts/boxes these days.Struggling in and out from the space-limiting area under the desk was what caused me to "lazy-out" with crawling out to get a flashlight. One improvement I will make in the near future is to install "permanent" lights under my table. Some light with an ON/OFF switch under the desk. So, next year (I hope), instead of lying to myself that I can see in limited light, I can flip a switch and really see.Quote from: Kilrah on January 07, 2020, 11:40:02 amNaw... the two blown drives were SATA/600mb, but powered via a molex-to-SATA power converter+splitter.I do have a few molex+IDE drives with (very old) backups on them. I have not touch them for years. I wonder if those drives can still spin. Perhaps I should check on those backups and move them to a newer drive...Enough break time - I do still have a lot to do because of the two dead drives (lots of juggling around to make sure critical files on server has a second copy even with 2 dead backup drives. Now I have to juggle them back to their proper places.)One "improvement" I already made - I will have at least two stand-by drives instead of just one - after I finish un-juggling the files.

author: wraper
author_rank: Unknown
timestamp: January 08, 2020, 05:08:22 pm
content: Quote from: David Hess on January 07, 2020, 03:58:45 pmHow it's a crutch? Please explain how otherwise SSD should know that deleted (but not erased) data is no longer in use and sector can be erased for further use? Not only that, if you try to partially rewrite such (non used) physical sector (as logical sector is smaller than physical), it will cause read/modify/ and write of garbage data to another physical sector. Causing write amplification and further performance reduction besides not being able to erase non used sectors in advance. And physical sectors are quite large, usually 512kB. IMHO you have no idea about SSDs.QuoteBSQuoteDunno what was your problem was but IMHO you made very big conclusions with very limited amount of knowledge about what actually happened.

author: Kilrah
author_rank: Unknown
timestamp: January 08, 2020, 05:35:41 pm
content: Quote from: wraper on January 08, 2020, 05:08:22 pmBy understanding the actual filesystem, but since it's impractical/impossible to design a drive that could understand every possible filesystem out there it's not a crutch but the only viable solution.Quote from: wraper on January 08, 2020, 05:08:22 pmThis however is correct. At some point every flash page has been used once and only TRIM can mark some as truly empty again, and that point comes quickly since the drive will typically favor using empty pages first before starting to do read-modify-writes on partially full ones.

author: David Hess
author_rank: Super Contributor
timestamp: January 09, 2020, 03:31:23 am
content: Quote from: wraper on January 08, 2020, 05:08:22 pmThen how did drives work before TRIM became available? How do they work if the OS does not use TRIM?The reserved space on the drive should provide plenty of space to absorb the difference between the virtual and physical sector mapping as old sectors are marked unused after being rewritten. In practice the performance advantage of TRIM only becomes significant with drives which were poorly designed in the first place with insufficient reserve space. It does make for great benchmarks and talking points though.QuoteOr it could be because I have tested various Flash media over the past decade and have an intimate understanding of how Flash media works at the hardware level.So tell me, why would a new SSD which has never been used report bad sectors?

author: wraper
author_rank: Unknown
timestamp: January 09, 2020, 09:52:52 am
content: Quote from: Kilrah on January 08, 2020, 05:35:41 pmAny modern SSD has some reserved amount of storage above what is visible to OS. Also they have garbage collection. Though it does not work too well without TRIM. As it only deals with remaining data of physical blocks where majority of their content was written to other physical blocks, no idea about deleted data. Thus moving a lot of garbage data around.

author: wraper
author_rank: Unknown
timestamp: January 09, 2020, 10:25:37 am
content: Quote from: David Hess on January 09, 2020, 03:31:23 amMaybe because they were really bad/ drive was faulty? Why drive would mark them bad if it was simply a retention problem? Also you did not say how many bad sectors.QuoteReserved area not visible to OS + working like slow crap after a while if there is no garbage collection implemented like in early SSD.QuoteIn practice give or take erasing and then writing is very slow compared to simply writing. Without TRIM the only way do deal with that decently is either understanding file system or having large amount of reserved area. But you still will get writing amplification as any garbage collection without TRIM is still moving orphaned data = excessive NAND wear. The only data it knows to be really orphaned is which was written over by OS (written to another block internally). Also reserved area may fill real fast when you try writing large amount of data resulting in abysmal performance before garbage collection is given idle time to do it's job. It would be proper to say that garbage collection without TRIM is the real crutch.

author: Jeroen3
author_rank: Super Contributor
timestamp: January 09, 2020, 10:50:12 am
content: Quote from: David Hess on January 09, 2020, 03:31:23 amRead-Erase-Write. On spinning disks there was no erase step*, just overwrite.It's especially bad when the erase blocks are misaligned with filesystem allocation units.The magic in an SSD is to only havewriteuntil all blocks are used.https://www.micron.com/-/media/client/global/documents/products/technical-marketing-brief/client_vs_enterprise_performance_use_cases_tech_brief.pdf?la=en*today we haveSMR, it's crap.