Placing a single file bits contiguously on a HD - Page 1
Forum section: electronics / projects

author: ricko_uk
author_rank: Super Contributor
timestamp: June 21, 2021, 06:04:49 pm
content: Hi,just trying some experiments and am wondering if there is a way to guarantee that a single file can be guaranteed to be stored on a HD so that every bit of the file is physically next to the previous one without any interruption. The additional assumptions here are:- the HD is the classic HDD/disk type NOT the SSD or any other memory type- the HD is brand new or empty and freshly formatted- preferably using FAT or exFATThere are two cases:1) if the file size is less than the sector size when formatted the HDD I assume the file is guaranteed to be stored in a single sector? Or can it still be split across multiple ones?2) if the file is considerably larger than the sector size (for example 10 or 100 times the size of the sector size) how can it be guaranteed that every single bit is physically placed next to the previous one on the drive?3) is there a way to specify and/or see the physical position of a byte or chunk of bytes on the drive itself? Is there any software that lets you see each byte and the physical position on the disk?Thank you as always!

author: retiredfeline
author_rank: Frequent Contributor
timestamp: June 21, 2021, 06:25:05 pm
content: Why do you even want to do this? You can't tell if the drive controller has encoded the block in such a way that the bits are enhanced with error detection/correction and stored in some scrambled order.

author: ledtester
author_rank: Super Contributor
timestamp: June 21, 2021, 06:42:19 pm
content: QuoteThis is a safe assumption.QuoteBe aware that the OS is balancing other concerns when it picks blocks for your file -- such as fragmentation. Also, hard drives may doing some remapping -- e.g. to avoid bad blocks.You can use a low-level interface to write data directly to specific blocks on the drive, but then you'll also be responsible for the integrity of the file system directory blocks.Programs which "defragmented" your disk by rearranging where files reside were a thing back in the 80s and 90s, but operating systems have gotten a lot better at file allocation since then.Another thing that modern OS's will do is to pre-fetch data from a file so by the time your program requests the next block from a file it's already in memory.

author: ledtester
author_rank: Super Contributor
timestamp: June 21, 2021, 06:56:22 pm
content: QuoteI'd look for "disk editor" / "drive editor" or "data recovery" program such as-http://www.winhex.com/disk-editor.html-http://www.runtime.org/diskexplorer.htmSuch programs are used for computer forensics -- e.g. to reconstruct deleted files and directories -- and provide a way to decode the information in the file system's directory blocks.

author: T3sl4co1l
author_rank: Super Contributor
timestamp: June 21, 2021, 08:05:26 pm
content: In FAT specifically, contiguous sectors form clusters, so you are guaranteed up to that size at least.Again, this is logical to the filesystem driver, and need not be true in the drive. It will be so, in simple enough (usually old) devices, uh not that vintage HDDs are probably all that good of an idea to use, but you can certainly get Flash chips of this type. (As for specific packages, I don't know; industrial CF or SD perhaps can? Something about NAND vs. NOR Flash as well.)Tim

author: Manul
author_rank: Super Contributor
timestamp: June 21, 2021, 09:24:44 pm
content: A lot of things about disk geometry, data placement can be analyzed by what is called microbenchmarking. Here is a good read about such experiments.https://blog.stuffedcow.net/2019/09/hard-disk-geometry-microbenchmarking/

author: magic
author_rank: Super Contributor
timestamp: June 21, 2021, 10:37:20 pm
content: Quote from: ricko_uk on June 21, 2021, 06:04:49 pmNope. Firstly, each sector is followed by its corresponding ECC checksum. Secondly, I'm not even sure if it's guaranteed that data within one sector are encoded as the exact sequence of bits you wrote or perhaps transformed. It's known that at least some SSDs perform scrambling.That being said, I have no idea what kind of experiment that you can reasonably perform could possibly reveal any difference here.For throughput, it only matters that consecutive blocks of file are written to consecutive sectors. That is easily done on a freshly formatted FS and a reasonable OS will store long files sequentially, or at worst, in large sequential chunks. You will get >99% maximum throughput of your disk.A single file can also be written onto disk directly, without any filesystem. Particularly easy on Unix.

author: viperidae
author_rank: Frequent Contributor
timestamp: June 22, 2021, 07:09:08 am
content: The only thing you can guarantee is the logical placement. The hard drive firmware determines physical location. It may be transparently reordering block and may not be writing consequitive blocks on the same platter. The blocks themselves could also be split across platters.From an application software point of view, you have the filesystem to contend with as well. While most filesystems allow applications to preallocate the storage for an entire file, the sequence of the blocks is still not guaranteed.

author: mariush
author_rank: Super Contributor
timestamp: June 22, 2021, 07:24:09 am
content: NTFS file system may not even reserve a sector for a file, if the amount of data is too small.I don't know the amount from memory, it's something like less than 155 bytes if I remember correctly, basically if the file size is less than this, there's a high chance the data will be in the file table where the metadata is ( file name, dates modified, created, last accessed, security descriptors/permissions, owner etc)Seehttps://web.archive.org/web/20210506084930/https://docs.microsoft.com/en-us/archive/blogs/askcore/the-four-stages-of-ntfs-file-growth- the original link is still live, but the embedded pictures are gone, so linking to recent copy (6th of may) that still has the images.You can also Windows API functions to preallocate disk space to a file, but I'm not sure it guarantees ONE continuous block of sectors ... I think it tries as best as possible to get big empty blocks.See SetFilePointerEx() and SetEndOfFile() and the Set­File­Information­By­Handle functions ... here's a couple links:How can I preallocate disk space for a file without it being reported as readable? :https://devblogs.microsoft.com/oldnewthing/20160714-00/?p=93875First answer :https://stackoverflow.com/questions/7970333/how-do-you-pre-allocate-space-for-a-file-in-c-c-on-windowsFAT32 is probably simpler but I'm not sure you're guaranteed to get a continuous chunk for your file... With NTFS you could probably reserve it some way, not sure there's an API for it ... the only other way I can think of is to work like a defragmenting tool and get direct access to the drive and find the largest unused area to create your file.Yeah ... and mechanical drives CAN reallocate sectors, if they become too unreliable. As for SSDs ... they're by design not continuous, because data is shuffled around to extend the life of the ssd.

author: fordem
author_rank: Regular Contributor
timestamp: June 22, 2021, 07:10:18 pm
content: Quote from: ricko_uk on June 21, 2021, 06:04:49 pmLet's define "every bit of the file". Do you mean every piece of the file or do you mean every binary digit, as in eight bits make a byte.

author: gabiz_ro
author_rank: Regular Contributor
timestamp: June 23, 2021, 05:31:14 am
content: You may try wincontig.

author: ricko_uk
author_rank: Super Contributor
timestamp: June 23, 2021, 02:43:12 pm
content: Thank you all for the detailed infos and apps suggested!