[solved] weird behavior with an ext4 filesystem - Page 1
Forum section: electronics / general

author: legacy
author_rank: Super Contributor
timestamp: March 16, 2019, 09:53:48 pm
content: so I have a 4GB microdrive, and I am trying to populate it with filesCode:mkfs.ext4 /dev/sda3mke2fs 1.43.9 (8-Feb-2018)Creating filesystem with 968562 4k blocks and 242400 inodesFilesystem UUID: ba8d7da1-7ad5-451a-97a3-ee7de5228d98Superblock backups stored on blocks:32768, 98304, 163840, 229376, 294912, 819200, 884736Allocating group tables: doneWriting inode tables: doneCreating journal (16384 blocks): doneWriting superblocks and filesystem accounting information .. doneCode:tar -xzf pack-01.tgz -C /mnt/disk2/No space left on deviceIt says the disk is full, but it's not the truth since the disk still has free spaceCode:df | grep sda3/dev/sda3 3747808 2694608 843104 77% /mnt/disk233% of space still available, let's check itCode:dd if=/dev/zero of=raw bs=1024dd: writing `raw': No space left on device1036813+0 records in1036812+0 records out1061695488 bytes (1.1 GB) copied, 1184.42 s, 896 kB/sdf | grep sda3/dev/sda3 3747808 3731424 0 100% /mnt/disk2the disk is not really full!so, it sounds it's possible to allocate the space for a singular file, but not for a multitude of files.This should have something to deal with block-size?Any idea?

author: legacy
author_rank: Super Contributor
timestamp: March 16, 2019, 09:59:38 pm
content: Code:tune2fs -l /dev/sda3 | grep -i 'block size'Block size: 4096

author: magic
author_rank: Super Contributor
timestamp: March 16, 2019, 10:01:23 pm
content: Probably some bullshit like reserved space (only root can allocate it) or the FS run out of inodes (only 242400 files can exist on yours). Both can be checked with tune2fs I think and overridden with mkfs flags.

author: legacy
author_rank: Super Contributor
timestamp: March 16, 2019, 10:06:28 pm
content: Quote from: magic on March 16, 2019, 10:01:23 pmUID=0, rootand there is no quotaQuoteI can only override the block-size. Ext3 and Ext4 use block sizes of { 1024, 2048, 4096 } bytes. The required size is chosen when setting up the file system with the mke2fs formatting program of Ext3/4. In theory, 4 KB is the maximum, and this block size corresponds to that of the kernel's memory pages in RAM, which makes paging easier for the operating system.

author: wilfred
author_rank: Super Contributor
timestamp: March 16, 2019, 11:41:17 pm
content: You can use "sudo df -i /" to check you haven't run out of inodes.How much space does your archive expand out to on a disk with ample space?

author: legacy
author_rank: Super Contributor
timestamp: March 17, 2019, 12:39:16 am
content: Quote from: wilfred on March 16, 2019, 11:41:17 pm2.8Gbyte of 3.7GB available

author: legacy
author_rank: Super Contributor
timestamp: March 17, 2019, 12:41:17 am
content: Code:mkfs.ext4 -b 1024 /dev/sda3mke2fs 1.41.14 (22-Dec-2010)Filesystem label=OS type: LinuxBlock size=1024 (log=0)Fragment size=1024 (log=0)Stride=0 blocks, Stripe width=0 blocks242176 inodes, 3874248 blocks193712 blocks (5.00%) reserved for the super userFirst data block=1Maximum filesystem blocks=71041024473 block groups8192 blocks per group, 8192 fragments per group512 inodes per groupSuperblock backups stored on blocks:8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409, 663553,1024001, 1990657, 2809857so, now the block-size is 1Kbyte and there are 242176 inodes ... let's check if it's ok

author: Nominal Animal
author_rank: Super Contributor
timestamp: March 17, 2019, 02:39:40 am
content: How many files and directories doespack-01.tgzhave? Say,Code:tar -tzf pack-01.tgz | wc -lIf that is over 200,000 you are definitely running out of inodes. For filesystems with lots of small files, I recommendCode:mkfs.ext4 -b 4096 -i 4096 -m 0 /dev/sda3The-iparameter controls the number of inodes (bytes-per-inode), and having it the same size as the block size allows the maximum number of "files". You can also specify the exact number of inodes using the-Nparameter.-m 0disables root-reserved blocks.The number of inodes really matters for small filesystems, and filesystems used at near full capacity. For example, on mail servers with Maildir mailboxes (useful with IMAP clients!), you definitely want to allow the maximum number of small files, as each message will be a separate file. A web server tends to have slightly larger files (so fewer compared to the filesystem size). A file/media server tends to have large files. It is up to the system administrator to choose that setting. The default is pretty good, but-b SIZE -i SIZEis technically the safest choice, because it allows the maximum number of "files" on said filesystem. (The reason that is not the default, is that each potential inode reserves 128 or 256 bytes on disk; so, 200,000 inodes takes up 25.6 or 51.2 megabytes of disk space, permanently. Halving that, via-b SIZE -i SIZE/2, halves both the maximum number of inodes/"files" and the permanently used space.)If you do not intend to use extended attributes (xattr, orACLs beyond the standard Unix ones), you can also use-I 128to use specifically smaller inodes. In this case, themountoptionsext_attr,acl, anduser_xattrmay not be available.So, in your particular case, I suggest you tryCode:mkfs.ext4 -b 4096 -i 4096 -I 128 -m 0 /dev/sda3

author: ejeffrey
author_rank: Super Contributor
timestamp: March 17, 2019, 02:50:30 am
content: Yes, running out of inodes is a classic problem with small files on traditional Unix filesystems.You can allocate more inodes at mkfs time with the -i flag or you can use -t to select a "use case" option that sets defaults.Or you can use xfs or btrfs that don't have statically assigned inodes. One problem with ext4 is that inodes are now 256 bytes (compared to a default of 128 in ext3) so if you use -i 4096 you will actually waste a lot of space for the common case.

author: legacy
author_rank: Super Contributor
timestamp: March 17, 2019, 09:37:51 am
content: Code:df -iFilesystem Inodes IUsed IFree IUse% Mounted on/dev/sda3 242176 242176 0 100% /mnt/disk2yup, Ifree is equal to zero

author: legacy
author_rank: Super Contributor
timestamp: March 17, 2019, 09:47:08 am
content: Quote from: Nominal Animal on March 17, 2019, 02:39:40 amCode:mkfs.ext4dev -b 1024 /dev/sda3 --> 242.176 inodesmkfs.ext4dev -b 2048 /dev/sda3 --> 242.760 inodesmkfs.ext4dev -b 4096 /dev/sda3 --> 242.400 inodesmkfs.ext2 -b 1024 /dev/sda3 --> 242.176 inodesmkfs.ext4 -b 4096 -i 4096 -I 128 -m 0 /dev/sda3 --> 968.640 inodesmore inodesI don't need { xattr, ACLs, etc }.

author: legacy
author_rank: Super Contributor
timestamp: March 17, 2019, 09:53:21 am
content: Quote from: ejeffrey on March 17, 2019, 02:50:30 amxfs: maybebtrfs: no, because it uses too many resources (on a router) and it's not supported with old kernels (2.6.37).now we are on kernel v4.9.6, because it solves a lot of quirks with the miniPCI (and managing quirks is easier on kernel v4.* than on kernel 2.6.*), but we ought to be supporting old kernels, so .. xfs is known to be working, while btrfs is a "new entry".anyway, xfs is on the way (the builder machine is compiling it), while I am going to try ext4 with the above setup

author: magic
author_rank: Super Contributor
timestamp: March 17, 2019, 10:07:12 am
content: Isn't btrfs that filesystem which still gets data corruption bugs every other month?ReiserFS could be more space efficient with tons of small files thanks to tail packing.

author: legacy
author_rank: Super Contributor
timestamp: March 17, 2019, 10:46:31 am
content: To summarize what we are doing: we have created a Qemu/MIPSEL machine to native compile stuff for mips1-le (superstrict subset patched to be compatible with everything, Ci20, Chinese MIPS, Atheros, rb532, etc...).it has been prepared with an old eabi2 portage, and it comes with a customized overlay with things that we are making specifically for our Sonoko project.At the end of the process we obtained a rootfs to be loaded on a physical machine. and of course, the little rb532 is the chosen one because it's the router we are using.Qemu can have and use a very large virtual hard drive with up to 20Gbyte of space, while the rb532 can only use microdrive (CF/Flash are not really compatible, due to a problem with the firmware) up to 6Gbyte, this due the compatibility with ATA-CF.Seagate did a couple of 8Gbyte, and even a 12Gbyte Microdrive units, which are also compatible with the ATA-CF spec, and released with a CF2 connector, but these things are rare and expensive.We currently have a stack of six 4Gbyte microdrive units, so this is the size of the disk layout we are aiming forCode:Disk: /dev/sdaSize: 3.8 GiB, 4095737856 bytes, 7999488 sectorsLabel: dos, identifier: 0x00000000Device Boot Start End Sectors Size Id Type>> /dev/sda1 63 16127 16065 7.9M 27 raw/dev/sda2 16128 250991 234864 114.7M 82 swap/dev/sda3 250992 7999487 7748496 3.7G 83 ext3Anyway, I have already changed the building rule of our-builder, so now I can part the big package into small packages, instead of obtaining a supermassive-all-packaged-together big file.Code:cyt restore --profile=small --list mips1-le-stage4-rb532.list --device=/dev/sda3formatting /dev/sda3:ext4 ... donerestoring mips1-le-stage4-rb532-bin ... donerestoring mips1-le-stage4-rb532-boot ... donerestoring mips1-le-stage4-rb532-dev ... donerestoring mips1-le-stage4-rb532-etc.my ... donerestoring mips1-le-stage4-rb532-etc ... donerestoring mips1-le-stage4-rb532-export ... donerestoring mips1-le-stage4-rb532-home ... donerestoring mips1-le-stage4-rb532-kernel ... donerestoring mips1-le-stage4-rb532-lib ... donerestoring mips1-le-stage4-rb532-mnt ... donerestoring mips1-le-stage4-rb532-opt ... donerestoring mips1-le-stage4-rb532-proc ... donerestoring mips1-le-stage4-rb532-proof ... donerestoring mips1-le-stage4-rb532-root ... donerestoring mips1-le-stage4-rb532-sbin ... donerestoring mips1-le-stage4-rb532-src ... donerestoring mips1-le-stage4-rb532-sys ... donerestoring mips1-le-stage4-rb532-tftproot ... donerestoring mips1-le-stage4-rb532-tmp ... donerestoring mips1-le-stage4-rb532-usr-bin ... donerestoring mips1-le-stage4-rb532-usr-include ... donerestoring mips1-le-stage4-rb532-usr-libexec ... donerestoring mips1-le-stage4-rb532-usr-lib ... donerestoring mips1-le-stage4-rb532-usr-man ... donerestoring mips1-le-stage4-rb532-usr-mipsel-unknown-linux-gnu ... donerestoring mips1-le-stage4-rb532-usr-overlay ... donerestoring mips1-le-stage4-rb532-usr-portage--eabi-2 ... donerestoring mips1-le-stage4-rb532-usr-portage ... donerestoring mips1-le-stage4-rb532-usr-sbin ... donerestoring mips1-le-stage4-rb532-usr-share ... donerestoring mips1-le-stage4-rb532-usr ... donerestoring mips1-le-stage4-rb532-usr-tmp ... donerestoring mips1-le-stage4-rb532-var ... donerestoring mips1-le-stage4-rb532-www ... doneThese files were the content of the pack-01.tgz, and are easier to be handledCode:cyt countfile --list mips1-le-stage4-rb532.listcounting mips1-le-stage4-rb532-bin ... 118 filescounting mips1-le-stage4-rb532-boot ... 8 filescounting mips1-le-stage4-rb532-dev ... 1557 filescounting mips1-le-stage4-rb532-etc ... 965 filescounting mips1-le-stage4-rb532-etc.my ... 225 filescounting mips1-le-stage4-rb532-export ... 5 filescounting mips1-le-stage4-rb532-home ... 2 filescounting mips1-le-stage4-rb532-kernel ... 3 filescounting mips1-le-stage4-rb532-lib ... 352 filescounting mips1-le-stage4-rb532-mnt ... 4 filescounting mips1-le-stage4-rb532-opt ... 2 filescounting mips1-le-stage4-rb532-proc ... 1 filescounting mips1-le-stage4-rb532-proof ... 3 filescounting mips1-le-stage4-rb532-root ... 818 filescounting mips1-le-stage4-rb532-sbin ... 160 filescounting mips1-le-stage4-rb532-src ... 41997 filescounting mips1-le-stage4-rb532-sys ... 2 filescounting mips1-le-stage4-rb532-tftproot ... 2 filescounting mips1-le-stage4-rb532-tmp ... 0 filescounting mips1-le-stage4-rb532-var ... 50745 filescounting mips1-le-stage4-rb532-www ... 2 filescounting mips1-le-stage4-rb532-usr-bin ... 925 filescounting mips1-le-stage4-rb532-usr-include ... 2351 filescounting mips1-le-stage4-rb532-usr-lib ... 12577 filescounting mips1-le-stage4-rb532-usr-libexec ... 155 filescounting mips1-le-stage4-rb532-usr-man ... 4 filescounting mips1-le-stage4-rb532-usr-mipsel-unknown-linux-gnu ... 61 filescounting mips1-le-stage4-rb532-usr-overlay ... 9542 filescounting mips1-le-stage4-rb532-usr-portage ... 103543 filescounting mips1-le-stage4-rb532-usr-portage--eabi-2 ... 1 filescounting mips1-le-stage4-rb532-usr-sbin ... 198 filescounting mips1-le-stage4-rb532-usr-share ... 23858 filescounting mips1-le-stage4-rb532-usr-tmp ... 0 filescounted 250186 filesQuote from: Nominal Animal on March 17, 2019, 02:39:40 amThe new implemented feature can answer this questionCode:counted 250186 files

author: legacy
author_rank: Super Contributor
timestamp: March 17, 2019, 10:59:14 am
content: Quote from: magic on March 17, 2019, 10:07:12 amGood idea! I have never used it, and it would be an interesting alternative. There is another little problem with the rb532: the kernel size must be inferior to 7.9Mbyte because the bootstrapping firmware is not able to manage a bigger raw-file form the first partition of the microdrive, neither it's able to manage a bigger file from the tftpboot.This was not a problem with kernel 2.6.*, but it's a big problem with kernel 4.* because they are really bloated.So I have to check how big the kernel 4.9.6 will go with- RiserFS as built-in only, by disabling { ext2, ext3, xfs } as built-in- xfs as built-in only, by disabling { ext2, ext3, RiserFS } as built-inNFS is compiled as a module, but the main filesystem needs to be compiled built-in.

author: legacy
author_rank: Super Contributor
timestamp: March 17, 2019, 11:28:51 am
content: Code:kanojo rb532-stage4-2019-03-15 # cyt restore --profile=small --list mips1-le-stage4-rb532.list --device=/dev/sda3 --verboseformatting /dev/sda3:ext4 {-b 4096 -i 4096 -I 128 -m 0} ... donemounting /dev/sda3 as /mnt/disk2 ... donerestoring mips1-le-stage4-rb532-bin ... donerestoring mips1-le-stage4-rb532-boot ... donerestoring mips1-le-stage4-rb532-dev ... donerestoring mips1-le-stage4-rb532-etc.my ... donerestoring mips1-le-stage4-rb532-etc ... donerestoring mips1-le-stage4-rb532-export ... donerestoring mips1-le-stage4-rb532-home ... donerestoring mips1-le-stage4-rb532-kernel ... donerestoring mips1-le-stage4-rb532-lib ... donerestoring mips1-le-stage4-rb532-mnt ... donerestoring mips1-le-stage4-rb532-opt ... donerestoring mips1-le-stage4-rb532-proc ... donerestoring mips1-le-stage4-rb532-proof ... donerestoring mips1-le-stage4-rb532-root ... donerestoring mips1-le-stage4-rb532-sbin ... donerestoring mips1-le-stage4-rb532-src ... donerestoring mips1-le-stage4-rb532-sys ... donerestoring mips1-le-stage4-rb532-tftproot ... donerestoring mips1-le-stage4-rb532-tmp ... donerestoring mips1-le-stage4-rb532-usr-bin ... donerestoring mips1-le-stage4-rb532-usr-include ... donerestoring mips1-le-stage4-rb532-usr-libexec ... donerestoring mips1-le-stage4-rb532-usr-lib ... donerestoring mips1-le-stage4-rb532-usr-man ... donerestoring mips1-le-stage4-rb532-usr-mipsel-unknown-linux-gnu ... donerestoring mips1-le-stage4-rb532-usr-overlay ... donerestoring mips1-le-stage4-rb532-usr-portage--eabi-2 ... donerestoring mips1-le-stage4-rb532-usr-portage ... donerestoring mips1-le-stage4-rb532-usr-sbin ... donerestoring mips1-le-stage4-rb532-usr-share ... donerestoring mips1-le-stage4-rb532-usr ... donerestoring mips1-le-stage4-rb532-usr-tmp ... donerestoring mips1-le-stage4-rb532-var ... donerestoring mips1-le-stage4-rb532-www ... doneunmounting /dev/sda3 ... donePerfect! It worked!Code:# mount /dev/sda3 /mnt/disk2# df -iFilesystem Inodes IUsed IFree IUse% Mounted on/dev/sda3 968640 247711 720929 26% /mnt/disk2

author: LapTop006
author_rank: Unknown
timestamp: March 17, 2019, 01:28:50 pm
content: Quote from: legacy on March 17, 2019, 09:53:21 amThe 2.6 series is more than old at this point, 3.0 came out in mid-2011.

author: legacy
author_rank: Super Contributor
timestamp: March 17, 2019, 01:50:56 pm
content: Quote from: LapTop006 on March 17, 2019, 01:28:50 pmkernel 3.* don't solve PCI quirks and introduce A LOT of work for our custom drivers that need to be rewritten specifically for a new kernel, and since we do not have the time for doing, we avoid switching to a new kernel until it's really necessary or until it really gives an enhancement.

author: ejeffrey
author_rank: Super Contributor
timestamp: March 18, 2019, 04:35:30 pm
content: Quote from: magic on March 17, 2019, 10:07:12 amI think btrfs is pretty stable these days. Probably more so than ReiserFS ever was. But definitely not an option if you are stuck supporting older kernels.But honestly I wouldn't actually recommend any of them for most applications. ext4 is the most stable and well tested linux FS, and while it isn't the most performant it doesn't have any glaring performance holes. Maybe the better way to say it is that it has similar performance characteristics to most "traditional" linux filesystems so the things that are slow are somewhat expected. Storage is to cheap to worry about storage efficiency. Just use ext4 and format it with the right options.I would only consider xfs or btrfs if ext4 just won't cut it. For instance if you need the COW or variable stripe raid of btrfs or if you need the 64 bit inodes of xfs (>2B files). If you are just trying to use storage 20% more efficiently don't bother. Again, with btrfs you need to be on recent kernels.I wouldn't recommend ResierFS under any circumstances.

author: legacy
author_rank: Super Contributor
timestamp: March 18, 2019, 04:44:42 pm
content: Quote from: ejeffrey on March 18, 2019, 04:35:30 pmcan you define recent kernels?hppa (workstation): we are on 4.16.2rs/p (home router): we are on 2.6.39rb532 (Sonoko project): we are on 4.9.6 <--- this is the machine the topic is indirectly talking aboutAkita (personal PDA): we are on 2.6.23

author: magic
author_rank: Super Contributor
timestamp: March 18, 2019, 07:58:15 pm
content: Well, with two minutes of searching, from last summer:https://patchwork.kernel.org/patch/10568533/I don't know if dedupe is on by default, maybe it's a nothingburger. But it fits into the general pattern I perceive that btrfs issues just "keep happening".Can't comment on ReiserFS nowadays, maybe you are right. I certainly tried that barely maintained Reiser4 port once and managed to break the FS within half an hour, but that's a different animal.

author: legacy
author_rank: Super Contributor
timestamp: March 18, 2019, 08:21:52 pm
content: so btrfs will be considered for any real projects starting from kernel >= v5.1

author: legacy
author_rank: Super Contributor
timestamp: March 24, 2019, 09:53:52 am
content: for comparing reasons, we compiled the same kernel with different filesystem supportfilesystem kernel sizeext4, 7.746.092ext2 + ext4, 7.791.414xfs, 7.992.833raiser, 7.551.430kernel 4.9.1