Disk usage discrepancy - Linux - Page 1
Forum section: electronics / general

author: Circlotron
author_rank: Super Contributor
timestamp: November 06, 2023, 10:04:43 am
content: I have a 1TB HDD and the used space figure is WAY different to the used on disk space figure. Also when using the Control Centre to look at Disks it gives a significantly different figure again. What could be happening?

author: magic
author_rank: Super Contributor
timestamp: November 06, 2023, 10:33:22 am
content: One thing is the idiotic "binary" versus "decimal" gigabytes issue: traditionally 1KB was considered 1024 bytes and 1GB was 1024³, which is some weird number somewhat larger than 10⁹. This likely accounts for the difference in overall disk capacity reported by different tools. HDD manufacturers used the binary convention until about 40GB, then switched to decimal in order to sell smaller disks at the same advertised capacity. At some point Apple and Microsoft did the switch too, but on Linux it's a mixup. The tool which reports 1.0TB uses the decimal convention which manufacturers use today, the other does not.But there is still remaining a difference in percentage of free space, about 5%. I'm not entirely sure, but this is an EXT4 filesystem and 5% is the exact amount of space which is "reverved" by default, i.e. only the root user can create new files or expand existing ones when total free space falls below 5%. It's possible that one tool doesn't consider this space free, while the other doesYou can check for this by running (as root):Code:tune2fs -l /dev/sdband looking for information about reserved blocks and block size (most likely 1 block is 4KB = 4096B).To disable reserved blocks altogether, runCode:tune2fs -r 0 /dev/sdbThe filesystem may need to be unmounted for this, not sure.

author: Nusa
author_rank: Super Contributor
timestamp: November 06, 2023, 11:17:00 am
content: The disk report is using decimal numbers. 1 TB is spelled out in bytes (actually a smidge larger than). 1 TB - 448 GB = 552 GB. 552/1000 = 55.2%The folder report is using binary numbers. 1 TB decimal (10*12, with 1000 GB per TB) = 0.9095 TB binary (2**40 with 1024 GB per TB)magic is right the numbers don't quite match up, but they're in the ballpark for overall disk usage. Perhaps there's a trash folder full of not-completely-deleted stuff? Or some of the thoughts magic mentions.The 130.5 GB number, that's folder specific, while the rest of the page is for the whole disk. The "on disk" number looks the same, referring to the space taken by that folder on the disk, usually only significantly different if there are many files that are smaller than the block size. In this case, they're close enough they round to the same number.

author: DiTBho
author_rank: Super Contributor
timestamp: November 06, 2023, 11:39:07 am
content: Code:# lsprettysize *138 Kbyte xtrans-1.2.5.tar.bz212 Kbyte xulrunner-1.9.2-patches-0.5.tar.bz214 Kbyte xulrunner-1.9.2-patches-0.9.tar.bz2628 Kbyte xvidcore-1.2.2.tar.bz294 Kbyte xvinfo-1.1.0.tar.bz2114 Kbyte xwd-1.0.3.tar.bz2107 Kbyte xwininfo-1.0.5.tar.bz2101 Kbyte xwud-1.0.2.tar.bz21013 Kbyte xz-4.999.9beta.tar.gz213 Kbyte yaboot-1.3.14.tar.gz63 Kbyte yacc-1.9.1.tar.Z1 Mbyte zip30.zip415 Kbyte zlib-1.2.3.tar.bz2680 Kbyte zziplib-0.13.58.tar.bz2years ago I developed a simple tool to list files this way.Internally, there are two functions:1) base10, so "1K" means "1000"2) base2, so "1K" means "1024"

author: Circlotron
author_rank: Super Contributor
timestamp: November 06, 2023, 11:58:06 am
content: Quote from: magic on November 06, 2023, 10:33:22 amRun that and I get:Code:tune2fs 1.44.1 (24-Mar-2018)tune2fs: Bad magic number in super-block while trying to open /dev/sdbFound a gpt partition table in /dev/sdbWhat is the significance of that?

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 06, 2023, 12:27:35 pm
content: Trytune2fs -l /dev/sdb1instead.(/dev/sdbrefers to the entire disk, including partition tables; and/dev/sdb1is its first partition, named on theDevice:line in the second screenshot.)

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 06, 2023, 12:40:36 pm
content: If we are precise, then1 kB = 1,000 bytes1 KiB = 1,024 bytes1 MB = 1,000,000 bytes1 MiB = 1,024 KiB = 1,048,576 bytes1 GB = 1,000,000,000 bytes1 GiB = 1,024 MiB = 1,048,576 KiB = 1,073,741,824 bytes1 TB = 1,000,000,000,000 bytes1 TiB = 1,024 GiB = 1,048,576 MiB = 1,073,741,824 KiB = 1,099,511,627,776 bytesper clause 4 of the ISO/IEC 80000-13:2008 standard. The names are silly ("kibibyte", "mebibyte", "gibibyte", "tebibyte"), but we can be precise and unambiguous if we want, nowadays.On the command line,df -hgives you a summary of your disk usage in powers often (kB, MB, GB, TB)two (KiB, MiB, GiB, TiB); for exampleFilesystem Size Used Avail Use% Mounted onudev 7.7G 0 7.7G 0% /devtmpfs 1.6G 1.7M 1.6G 1% /run/dev/mapper/nvme0n1p3_crypt 464G 350G 91G 80% /tmpfs 7.8G 0 7.8G 0% /dev/shmtmpfs 5.0M 4.0K 5.0M 1% /run/locktmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup/dev/nvme0n1p2 3.7G 445M 3.1G 13% /boot/dev/nvme0n1p1 749M 6.1M 743M 1% /boot/efitmpfs 1.6G 56K 1.6G 1% /run/user/1000On the other hand, use an upper-case H,df -Hand you get the summary in powers oftwo (KiB, MiB, GiB, TiB)ten (kB, MB, GB, TB):Filesystem Size Used Avail Use% Mounted onudev 8.3G 0 8.3G 0% /devtmpfs 1.7G 1.8M 1.7G 1% /run/dev/mapper/nvme0n1p3_crypt 499G 376G 98G 80% /tmpfs 8.4G 0 8.4G 0% /dev/shmtmpfs 5.3M 4.1k 5.3M 1% /run/locktmpfs 8.4G 0 8.4G 0% /sys/fs/cgroup/dev/nvme0n1p2 4.0G 467M 3.3G 13% /boot/dev/nvme0n1p1 785M 6.4M 779M 1% /boot/efitmpfs 1.7G 58k 1.7G 1% /run/user/1000Edited:kjpye noticedI got them swapped myself.For details, seeman 1 df.

author: Zero999
author_rank: Super Contributor
timestamp: November 06, 2023, 01:36:36 pm
content: The following question was asked at a pub quiz last week: "How many megabytes in a terabyte?"I told my team mate the quiz master didn't specify whether it was TB or TiB, but he didn't know the difference, so we answered 1024. Fortunately it was right, but if not we should have been able to argue it. Afterwards, he Googled it.

author: magic
author_rank: Super Contributor
timestamp: November 06, 2023, 01:52:11 pm
content: 1048576

author: eutectique
author_rank: Frequent Contributor
timestamp: November 06, 2023, 04:06:30 pm
content: Quote from: DiTBho on November 06, 2023, 11:39:07 amFromman ls:Code:-h, --human-readablewith -l and -s, print sizes like 1K 234M 2G etc.

author: DiTBho
author_rank: Super Contributor
timestamp: November 06, 2023, 05:12:57 pm
content: Quote from: eutectique on November 06, 2023, 04:06:30 pmI wrote a simplified version of "ls" because I some need formatting that isn't offered by ls by default, and I don't have any desire to hack "ls", since then I would have to fork an ebuild (ls is part of sys-apps/coreutils) into an Overlay for the only purpose of adding a patch to ls.Code:Portage: sys-apps/coreutilsBinary: /bin/lsCode:Overlay: dev-tool/lsprettysizeBinary: /usr/bin/lsprettysizeThat's annoying on Gentoo with complex ebuilds like sys-apps/coreutils, so I usually end up that I'd better create my own tools, especially because they take less time to compile.Anyway, the point is: even a customized and simplified "ls", I found thatthere are two ways to show the filesize!And the problem is how to avoid to make confusion with the base { 10, 2 } the Eng constants { K, M, G, ... } are referring to.Above there is a solution, names are a bit silly, but actually they offer a solution.

author: kjpye
author_rank: Contributor
timestamp: November 07, 2023, 02:22:30 am
content: Quote from: Nominal Animal on November 06, 2023, 12:40:36 pmAre you sure about that? On my system it's the other way around, and the numbers in your second example are larger than in the first example, and GiB and friends should give smaller numbers than GB and co.

author: IanB
author_rank: Super Contributor
timestamp: November 07, 2023, 02:56:43 am
content: What I don't see anyone mention is allocation size.In all file systems, the space is managed (allocated and freed) in blocks of a fixed size. For example, suppose the block size is 4K, and you create a file containing 100 bytes of data. Then the "file size" will be 100 bytes, but the size on disk will be 4 kilobytes.It follows that all files occupy more disk space than the actual data they contain.Take it to the extreme, and create 1000 files of size 1 byte. You will have 1000 bytes of data occupying 1000 x 4K = 4 MB of disk space. Of course, 4 MB is lost in the noise of a 1 TB drive, and so people don't really worry about this very much, especially since real files are much bigger than 1 byte. But it's worth knowing.When I was in high school, we had a time share service on a PDP/11 at a local college. A typical account shared by a whole computer science class was allocated 100 blocks of 512 bytes. So 100 files of size 1 byte would fill it up. While you were logged in, you could use much more space, but you could not log out until you had deleted files down to the 100 block limit.I wrote a compress/decompress program that would take all the files in the directory and pack them into one file like zip without compression. So assuming 6.3 file names, those 100 1-byte files could be compressed down to a single file of a little more than 1000 bytes, or 2 disk blocks instead of 100 disk blocks. The class teacher could therefore run compress before logging out, and run decompress when logging in for the next class. Of course, real files had more than a byte, but typical student BASIC programs were very small and had a lot of wasted allocation space. Something like 3:1 compression was not atypical.

author: IanB
author_rank: Super Contributor
timestamp: November 07, 2023, 03:08:19 am
content: Quote from: Nominal Animal on November 06, 2023, 12:40:36 pmI've got to say, that nobody is forced to adopt published standards. Standards find acceptance if they are useful and solve problems. If they create more problems than they solve, then nobody needs to use them.In this case, it was (and is) universally accepted in computer science that 1 KB = 1024 bytes, 1 MB = 1024 * 1024 bytes, and so on. The ISO "standard" has created ambiguity by leaving it uncertain whether 1 MB is 1024x1024 bytes, or 1000x1000 bytes. Standards are supposed to remove ambiguity, not create it.The so-called standard would have been much more useful if it had proposed new nomenclature for the decimal sizes, for example, 1 KeB = 1 000 bytes, 1 MeB = 1 000 000 bytes, and so on.Some people who are stuck in an insular worldview would be astonished to learn that 1 Mlb = 1000 pounds (not one million pounds, which would be 1 MMlb).

author: SiliconWizard
author_rank: Super Contributor
timestamp: November 07, 2023, 03:59:51 am
content: As Nominal said, I also find the names silly, but that's a standard and see no reason not to adopt it. I have.Now feel free not to adopt it, as long as you are explicit about your units in your documents/communication. But experience shows that people are not, precisely because they assume that it's all obvious, when obviously, it isn't, and is stil a source of endless confusion as this very thread shows.I have heard on a regular basis, from people against the standard, that it was all obvious from the context anyway. Like, talk about RAM size, it's obvious KB is KiB, right? Talk about storage size, and it's obvious that KB is 1000 bytes, right? Or maybe not? What about throughput in MB/s? Is it MiB/s or not?I understand that old habits are hard to beat, and when the new stuff sounds silly, it doesn't help, but really, when units depend on context to have a full meaning, then it's not science or engineering, it's just literature.

author: DiTBho
author_rank: Super Contributor
timestamp: November 07, 2023, 04:37:09 am
content: in my lsprettysize tool, I just added a legenda ( --help ) to tell the user about the base { base10, base2 }.

author: IanB
author_rank: Super Contributor
timestamp: November 07, 2023, 04:55:33 am
content: Quote from: SiliconWizard on November 07, 2023, 03:59:51 amWhen a standard comes along and says that the thing that has always been 1024 bytes is now 1000 bytes, that is not a standard, that is a recipe for chaos.We can see the truth about the chaos from the endless arguments the attempted standard has produced. A good standard would produce harmony, not disruption.There was never any disagreement between memory space and disk space either. Reference my note above concerning disk allocation sizes. They were (and still are) in powers of two. From 512 byte blocks, to 1024 byte blocks, to 4096 byte blocks, and so on.

author: magic
author_rank: Super Contributor
timestamp: November 07, 2023, 07:40:26 am
content: Quote from: IanB on November 07, 2023, 02:56:43 amBecause it (hopefully) isn't the source of OP's confusion, but you have now added some more

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 07, 2023, 08:59:36 am
content: Quote from: kjpye on November 07, 2023, 02:22:30 amNo, I got them swapped.Seeman 1 dffor details;-his shorthand for--human-readable, and means powers-of-two scale, and-His shorthand for--si, meaning powers-of-ten scale.Will edit that post; thanks.

author: Circlotron
author_rank: Super Contributor
timestamp: November 07, 2023, 10:46:09 am
content: Quote from: magic on November 07, 2023, 07:40:26 amThis is the source of the confusion, the difference between these two figures.

author: tunk
author_rank: Super Contributor
timestamp: November 07, 2023, 03:55:07 pm
content: What does df say, and what linux distro?Do you have the same problem with the other disks?Just speculation, it could be a bug in the software.

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 07, 2023, 04:44:29 pm
content: Quote from: Circlotron on November 07, 2023, 10:46:09 amYou have 130.5GB in the root directory of the ONE_TB volume, and a total of 586.3GB overall in that volume.Let's say you have ONE_TB mounted in the typical location in Linux,/media/circlotron/ONE_TBand in it,/media/circlotron/ONE_TB/movie1.mkv:/media/circlotron/ONE_TB/movieN.mkvthat total to 130.5GB. Then, you have a subdirectory in it,/media/circlotron/ONE_TB/not-porn/movie1.mkv:/media/circlotron/ONE_TB/not-porn/movieN.mkvthat total to 455.8GB.Many Linux file managers only show you the folder statistics when you right-click and select properties. The summary is then of only that folder, not including any subfolders. Above, you have 586.3GB of stuff on a volume named ONE_TB, about 130.5GB in the topmost directory, the rest, about 455.8GB, somewhere in subdirectories.Me, I use Nemo right now, and that starts scanning the subdirectories interactively when you look at the properties of a folder, updating the total in real time as you wait. I don't know which file manager or desktop environment Circlotron is using, and there are several that can be configured to look like that, so it could also be a funky bug in one instead.

author: magic
author_rank: Super Contributor
timestamp: November 07, 2023, 05:00:47 pm
content: XFCE counts subdirectories too, and it takes quite a while on large disks for obvious reason.One thing that could still cause such recursive count to be wrong is lacking permissions to access some subdirectories.OTOH, I don't think the discrepancy can be completely (or even mostly) explained by the aforementioned issue of 4KB block allocation, because it would take a huge lot of small files to waste so much space in this manner.At the end of the day, I would trustdfto accurately report available space andduto accurately report overall disk usage of a directory (provided that it doesn't report failures due to lack of permissions). Meanwhile with those GUI tools you always have to guess what they are doing and what they might be doing wrong...

author: DiTBho
author_rank: Super Contributor
timestamp: November 07, 2023, 05:25:07 pm
content: my "lsprettysize" can be used like "du", but in this case it doesn't sum the actual size of the file but rather it sums how many blocks each file uses, so also including metadata blocks!This way, by iterating hierarchically from a sub-root, and adding up how many blocks are used in total, you get a better idea of how much real space you have consumed!

author: madires
author_rank: Super Contributor
timestamp: November 07, 2023, 05:40:56 pm
content: Another thing to know it that linux usually reserves 5% of a partition for root.

author: eutectique
author_rank: Frequent Contributor
timestamp: November 07, 2023, 05:43:13 pm
content: Quote from: magic on November 07, 2023, 05:00:47 pmYou don't need files, directories are equally wasteful. Here is an example -- 5 directories, the innermost contains a file of 1 byte.tree -hreports 1 byte for the file and 4K for each directory:Code:work> tree -h oneone└── [4.0K] two└── [4.0K] three└── [4.0K] four└── [4.0K] five└── [ 1] one-byte.txt4 directories, 1 filedu -shreports 24K allocated for the whole thing:Code:work> du -sh one24KoneXfce file manager (thunar) says "6 items, totalling 1 byte"

author: magic
author_rank: Super Contributor
timestamp: November 07, 2023, 06:40:37 pm
content: It would also take a huge lot of directories to waste 450GB with only 130GB of file dataBut good to know that XFCE also sucks.That's why I generally distrust such things.

author: DavidAlfa
author_rank: Super Contributor
timestamp: November 07, 2023, 07:39:52 pm
content: For some strange reason I still can't understand, some retard decided 2^10=1024 was discriminating the International System of Units or something like that.The all-time Megabyte (2^20 =1.048.576 Bytes) became 10^6=1.000.000 bytes, and the old measuring method was renamed to Mebibyte (MiB).Some file explorers report MiB, others MB, a file might be 12GiB or 11.17GB. It's a *** mess.This might have something to do with storage manufacturers, who (To the confused consumer) decided to sell devices using this measurement system.So, you got 1TB (1000*1000*1000*1000 bytes) but 931GiB. Still, 448GB would be 417GiB, the difference is too high.Try:Code:df -h

author: golden_labels
author_rank: Super Contributor
timestamp: November 07, 2023, 11:09:31 pm
content: The debate about prefixes, while the difference is 75%. Yes, that must be this new fangled Fibonacci unit. Fibobyte = 1650 bytes, fimobyte = 16502= 2722500 bytes, figobyte = 16503= 4492125000 bytes; 130.5 figobytes = 586222312500 bytes ≈ 586 GB!Circlotron: your file manager reports total size of files by counting them one by one. With insufficient access rights, it can’t read sizes of all files and they are not counted towards the total. So I reckon this is what’s happening. You are seeing incomplete value from counting, while the filesystem usage is reported directly from information about the filesystem and therefore contains the actual usage.Quote from: DavidAlfa on November 07, 2023, 07:39:52 pmHow binary prefixes came into existence would be a good substance for psychological studies. Starting from how an informal notation, used to indicate an order of magnitude, has became a part of a nerd lore, through how it turned into a collective false memory and invaded professional circles, finishing with fanatical reactions from people facing a mere suggestion the lore is false. And all this in just 30–40 years.

author: magic
author_rank: Super Contributor
timestamp: November 08, 2023, 07:06:16 am
content: If you bothered to read the thread you are responding to, you would understand that prefixes are responsible for one of the issues brought up by the OP.As for the rest of your post, aren't you some goddamn millenial kid anyway?

author: Circlotron
author_rank: Super Contributor
timestamp: November 09, 2023, 01:16:04 am
content: Quote from: Nominal Animal on November 07, 2023, 04:44:29 pmI'm using Mint Mate with Caja file manager.The disk has no files in the root directory. No hidden files either. Everything is in folders.For comparison, here is a disk on my work pc. The numbers are exactly as I would have expected, not like the other one.

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 09, 2023, 02:14:08 am
content: Quote from: Circlotron on November 09, 2023, 01:16:04 amAh, my mistake. The two differ even in the size of the device! Are you sure they mount the same partition? One definitely uses a standard ext3/ext4 mount, whereas the work pc mounts it using FUSE.Could you run and report the outputs ofsudo fdisk -l /dev/sdbwhich describes the partition table on the device itself (feel free to obfuscate disk model and identifier),df -H /media/$(id -un)/*which reports the space on any mounted external filesystems (using powers of ten kilo/mega/giga/tera suffixes), andsudo tune2fs -l /dev/sdb1which describes the properties of the first partition (assuming ext2/ext3/ext4 format)? They are both non-destructive. Of the latter, we're interested in "Filesystem state", "Inode count", "Block count", "Reserved block count", "Free blocks", "Free inodes", "Block size", and "Last checked".You cannot run a filesystem check while it is mounted, but if you unmount the volume first, you can then runsudo e2fsck -f -n /dev/sdb1which checks the filesystem without making any changes; if it finds any issues, consider re-running it without the-noption, in which case it asks whether you want it to try and fix issues it finds.

author: magic
author_rank: Super Contributor
timestamp: November 09, 2023, 06:57:35 am
content: I suppose these are different disks

author: Circlotron
author_rank: Super Contributor
timestamp: November 10, 2023, 02:57:56 am
content: Quote from: magic on November 09, 2023, 06:57:35 amAnd that is the point.On the second disk the numbers make sense, on the first disk they are confusing.

author: ejeffrey
author_rank: Super Contributor
timestamp: November 10, 2023, 03:41:40 am
content: Quote from: IanB on November 07, 2023, 03:08:19 amIt really wasn't. Despite frequent claims to the contrary and a notable lawsuit, manufacturers *never* consistently reported hard drive size with 1 megabyte = 1024*1024. Prior to the early 90s, manufacturers just did whatever they felt like, later they all adopted base 10 terminology. Most notably in the consumer space, the ST-506, *the* original consumer level hard drive came in 5 MB and 10 MB (formatted) sizes, 5,013,504 bytes and 10,027,008. Mainframe hard drives from the 70s were 100% exclusively power-of-10.And other uses were exclusively base 10, particularly communication. For instance 10 megabit ethernet was always 10*10^6 bits/second, nobody ever considered that it should be anything else.There were like 2 years in the late 80s where it was pretty common for *consumer* hard drives often used power-of-two sizes in their marketing materials. That just happens to correspond to when a lot of nerds of a certain age got their first hard drive, and led them to mistakenly believe that there was some universal standard until the marketing department figured they could use the confusion to inflate their product copy. But that's really not how it happened.It's also a bit of a silly convention, at least for hard drives. Once you get past the sector size of 512-4096 bytes there is no physical or logical reason to keep using binary powers. Drive platter counts and cylinder counts have no reason to be power-of-two aligned, and in fact usually aren't, nor are sector counts which aren't even constant.

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 10, 2023, 04:33:51 am
content: Quote from: Circlotron on November 07, 2023, 10:46:09 amAh. Me fail.I misunderstood. I do believe magic is right here.As Caja traverses the filesystem to obtain the amount of space used by the contents of the files (usingnftw()withfn()that sums the used space) in the textual summary, but the total space free and used for the pie chart via anstatvfs()call (that does no traversal, just peeks at the filesystem statistics), the significant difference is caused by Caja not havingreadaccess to all directories.Note that traversal ('x') access to all directories is required but not sufficient; Caja needs read ('r') access also, so that it can get the file information. It does not need to have read access to the files, though.Because theducommand-line command does the same traversal, it suffers from the same limitation, unless you temporarily elevate its privileges. You can verify this is indeed the case, by simply runningdu -h /media/$USER/ONE_TB >/dev/nullas it will tell you which directories it cannot read, and whose contents (including subdirectories) are therefore not counted in its summary.Caja's textual summary is limited in the exact same manner, it just doesn't visibly complain.

author: Circlotron
author_rank: Super Contributor
timestamp: November 10, 2023, 05:47:29 am
content: Quote from: Nominal Animal on November 10, 2023, 04:33:51 amI ran that command and it saidCode:du: cannot read directory '/media/myname/ONE_TB/.Trash-0': Permission denieddu: cannot read directory '/media/myname/ONE_TB/lost+found': Permission deniedTurns out in .Trash-0/expunged/ there was a folder with 53.7GB of files from an old disk image. They were meant to have been put in the trash but they don't show there. The file owner is root, not me. That still doesn't account for the whole discrepancy though.Edit-> opened Caja as root and looked in the trash folder on that disk and there is six folders with a total of 352GB of disk image files. So it seems the program I used to make the disk backups, Redo Rescue, had root permissions. Never thought of that. Oh well, I think that kind of solves it.

author: Circlotron
author_rank: Super Contributor
timestamp: November 10, 2023, 05:58:17 am
content: Not quite...Run Caja as root and try and delete the files and I get the following error message in the console.** (caja:5902): WARNING **: 16:56:17.701: Could not inhibit power management: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Name "org.gnome.SessionManager" does not exist

author: Nominal Animal
author_rank: Super Contributor
timestamp: November 10, 2023, 06:36:10 am
content: Quote from: Circlotron on November 10, 2023, 05:58:17 amYou can ignore that error. It occurs exactly because you're running Caja as root, as it cannot connect to your session manager and user dbus agent. All that warning means is that Caja could not connect to power management. (It only does so to ensure your machine does not go to sleep or hibernate while doing significant work via Caja, like copying files around.)If you want, you can runcd /media/yourself/ONE_TBsudo rm -rf .Trash-0sudo rm -rf lost+foundcdinstead. Obviously,sudo rm -rfis the nuclear weapon among Linux commands, allowing you to render your system completely inoperable, but the above pattern –– changing the working directory to the one containing the offending items, and then running the command to delete each one with one file or directory name only, no paths, no slashes! –– makes it much safer. Assuming thesudo rmcommands are written correctly, the worst thing that can happen is that you completely delete a similarly named tree elsewhere (if and only if thecdcommand failed or had a typo); in this case, these two are safe to run.lost+foundis a directory used to put orphan inodes and other similar oddities, when the filesystem is checked (usinge2fsck/fsck). Just be careful, and re-check the command before hitting Enter.The last "cd" alone switches back to your home directory, as you cannot unmount the device if you have a shell open with its working directory on that device (making such a mount "active"). In Linux, and more generally in BSDs and Unix, the current working directory for each process is not based on the path, but the actual directory itself. The kernel itself basically keeps a file description open on the directory. This means that if you in one shell run "cd ; mkdir example ; cd example" to create a subdirectory named "example" under your home directory, and then change the name of that subdirectory in another shell (cd ; mv example foobar), the first shell still works just fine and does not notice its actual path has changed, and is able to access everything in the renamed directory. The only thing that will fail is "cd $CWD" and similar commands that use the pathstring. This is also why mounts stay active (and not unmountable) as long as you have a shell open in there. It also means that with the abovesudo rmpattern, if you check you are in the correct directory by using e.g. "ls -laF" (to list the contents of the current working directory), nobody can do hidden tricks to cause you to delete the wrong things by the immediately followingsudo rmcommands. In some other operating systems, a nefarious user also logged in might be able to do that by renaming the directory you have a shell open, replacing it with a symlink to some other directory containing the thing they want you to delete, because they track paths using the string, and not the actual inodes. This kind of race window is not possible when using file description based approaches, only when paths are used.And that last one is the reason why I keep telling people that if they write code to traverse directories usingopendir()/readdir()/closedirinstead ofnftw()or FTS family of BSD functions, they will almost certainly open up bugs that would allow similar bait-and-hook path-based trickery to work. The underlying file description based machinery grew the entireATFILEstuff, including functions likeopenat(),linkat(),unlinkat()(actual modern syscall used to delete files and directories), and evenexecveat(), to protect against path manipulation during operations. Simply put, these, including current working directory, don't care if thenameused to access the directory or any of the parent directories changes; they have a robust name-bypassing "hook" into the actual directory or file instead.If youdowrite your own directory traversal code, then you should definitely be usingopenat()andfstatat(), or your code IS vulnerable to path bait-and-switch attacks and bugs, including simple file renames within the same directory. (The trick withopendir()in Linux is to use/proc/self/fd/NwhereNis the nonnegative file descriptor to a read-only descriptor opened byopenat()to the desired subdirectory.) A well-implemented directory traverser maintains a global set of device,inode tuples identifying each directory having already traversed, and while traversing each directory, a device,inode tuple for each file. It can still miss files renamed during the traversal (but usually does not, because renames-in-place tends to not reorder the directory contents), but that is acceptable; it will be able to deal with directory renaming during traversal, and not report both pre-rename and post-rename names and statistics for files that are renamed or moved during traversal. As you can see, that takes quite a lot of code, so it is better to rely on thenftw()instead provided by the standard C library (POSIX feature) instead.Apologies for the bit of a rant, but itispertinent to the discussion at hand. I bet not even DiTBho has done the traversalcorrectlyin their "lsprettysize" program, and I just hate shoddily done, known-vulnerable base utilities.

author: IanB
author_rank: Super Contributor
timestamp: November 10, 2023, 08:24:47 am
content: Quote from: ejeffrey on November 10, 2023, 03:41:40 amI do not concur with this, since total hard drive size is not really of particular significance. What is of much more relevance is operating systems, software, memory management, file management systems, and the size of files stored on the disk. In that area, memory is usually organized in pages sized in powers of two, and disk storage is measured in blocks which are also sized in powers of two for convenience of memory mapping and paging. This has been true from early on right up to today.For instance, minicomputers like the DEC PDP/11 or VAX machines measured disk storage in blocks of 512 bytes. The Multics mainframe system measured disk storage in pages of 1024 words, and segments of up to 64 pages. This goes back to the 70's, long before the 90's came around.

author: magic
author_rank: Super Contributor
timestamp: November 10, 2023, 08:46:49 am
content: Quote from: ejeffrey on November 10, 2023, 03:41:40 amSome of those nerds still have their first disks.My "1 gigabyte" Seagate ST51080A is labeled "2,116,800 sectors, 1083 Mbytes", which is indeed 1083·10⁶ bytes, but it notably isn't 10⁹ bytes and rather happens to be 1.009GiB. They were also quite careful to avoid saying "megabytes" or "MB". This disk was made in 1996 and things stayed that way through the '90s.As for universal standards, let's just agree that the universal standard of 1990s consumer computing was Microsoft Windows. In Windows, 1MB stood for 1024KB and 1GB for 1024MB, and that's why disks were made this way. And I wouldn't be surprised if Windows wasn't the only major OS to do so. Linux, for one, also used the same convention.So maybe it's more fair to to talk about a battle between marketing and consumer expectations which lasted for two decades before the marketers ultimately won, but you can't say that nothing happened.

author: golden_labels
author_rank: Super Contributor
timestamp: November 11, 2023, 07:15:06 pm
content: Circlotron:did you read my response?Quote from: ejeffrey on November 10, 2023, 03:41:40 amThe even funnier part is that RAM and RAM sticks manufacturers also use 10-based prefixes for everything except capacity. A 19200 MB/s DDR4 stick is 19.2·106B/s, not 20132659200 B/s (19200 MiB/s).

author: Circlotron
author_rank: Super Contributor
timestamp: November 12, 2023, 10:14:14 am
content: Quote from: Nominal Animal on November 10, 2023, 06:36:10 amOkay then. That got things moving a bit! There is still a remaining difference that I can't account for though. Maybe I'll fire up the Redo Restore CD and see if some extra files become visible.

author: magic
author_rank: Super Contributor
timestamp: November 12, 2023, 11:10:25 am
content: If you haven't already, rundf -hto get reliable usage statistics. Compare with what you see above.Runduon the mount point directory again to get reliable count of total disk usage. If it still reports inaccessible files, delete them too or runduas root.The resulting numbers should be close to each other. Then you can try to guess what your file manager is doing.

author: Circlotron
author_rank: Super Contributor
timestamp: November 12, 2023, 12:13:14 pm
content: df -h says 917G size 122G used 749G freedu says 127445556 right at the bottom, so 127G more or less.Those two figures are near enough to 130.5GB, so why do we have "180.5GB used"And why would the pic above now say "free space unknown" ?

author: magic
author_rank: Super Contributor
timestamp: November 12, 2023, 12:50:23 pm
content: These numbers check out becausedureports in 1KiB blocks by default (I forgot that it too has an -h option which switches to MiB/GiB automatically) and 127445556KiB is 121.5GiB.This is exactly the 130.5GB shown by your file manager, which appears to be using the decimal convention.Similarly, 984GB = 917GiB, so total capacity matches too.Free space also matches - 804GB = 749GiB.At this point we must notice that "used"+"free" doesn't add up to "size" indfoutput. I strongly suspect that most of the difference is the reserved space I mentioned in the first reply (see Nominal's reply as to why the command I posted didn't work, and what to do instead). Some of the difference may also be filesystem metadata.