Change:
I want to start 100 identical containers on NixOS. I have already raised the inotify maxuserinstances limit like suggested in That helped to get 40 containers running, but 100 explode shortly after start. Here are the log lines from a single container: then it's trapped in some loop. the same lines over and over again... I see no reason for this. I think there are still limits present that needs to be raised. Hardware should be no problem since i have tested it on a 16 core Xeon and 48 GB RAM computer! See the full syslog: (25MB) Build ISO image and create bootable USB drive. You might need 25 GB RAM or a lot of SWAP! Then just boot from the USB drive! (everything in ramdisk, computer will remain untouched) system: host os: multi-user?: sandbox: version: channels(root): nixpkgs:
Configure the containers to not autostart and start them manually, with some delay! I this to my , so it starts automatically: To fix this, we have to tell systemd to not overload the system while starting services (containers). I'm not sure how. Current unit file of first container: from Maybe systemd should only start so many containers as the system has core and wait then until they are ready. Or just 1. Or make it an option with conservative setting 1, but users can experiment how many their system can start concurrently.
The configuration happens in Any idea Maybe upstream has a solution for that. We are not the only people using systemd-nspawn.
Hey 100 identical containers seems a bit overkill to me. You will get a lot of processes that are just container boilerplate without any advantages. Normally containers are used for network-separation (thats why I use nixos/systemd containers), dependency-separation (the whole docker world is coming from that) or user-/customer-separation (which is the only reason I see for 100 identical containers). Gladly the dependency-separation scenario is being take care of in nixos already… I have much less containers in use, but with more heavy 'payloads'. And when all are started at the same time, I get the same problems. I define dependencies between containers like via systemd's property. That is a loose coupling so when both containers are started at the same time, they start sequentially, but when one fails and is restarted, the other is not restarted (as opposed to afair). Another way (what I want to explore 'when I have time'): With containers a lot of very similar boiler-plate processes are running just to get different PID-/user-/network-/filesystem-namespaces set up. I think the same can be reached by configuring these things for systemd units without the need of a systemd and journald and … inside the container.
my usecase: my company develops a Vulnerability scanner, some customers have hosts to scan. i work in QA and want to build a test environment with as many targets (containers with some services and an IP) as possible with not so expensive hardware. We scan only 10 targets at a time, so only 10 containers are really active. The others just idle. So i think when the host can handle 10 containers, it should be able to handle as well (from the cpu load).
I agree that this should work. I also think however that it's probably not a problem at our side; given that nixos-container is a thin wrapper around nspawn. Can you reproduce this bug with plain nspawn containers?
Can you reproduce this bug with plain nspawn containers? That would be a huge effort to setup. In NixOS it's just a few lines :) Since the problem is that systemd start too many things at the same time, they should provide a solution. It's maybe not nspawn related, just plain systemd unit config. I'm not an expert in that.
The workaround sounds not too complex to me, given it's quite an extreme situation IMHO. (It could be some automatically executed script.)
yes, we could just put a sleep in the container start script, but the time needed depends on the container. and since systemd knows when a container is ready, it should be able to coordinate it better. that should be a basic feature of systemd... For my usecase, i could just execute that as a script at start.
I would start by trying to figure what exactly is failing. This is not apparent from the logs (the only things that jump out are and ). You may want to strace and its children.
still appears with the workaround, so probably not the reason for the issue. Same with . When comparing the log of the failing system and the working one, most lines are identical. The relevant information seem to be: After that, systemd starts to kill container processes! For comparison the log on the working system with workaround: I don't see anything helpful in output, but here it is:
This looks like a solution: TimeoutStartSec is the default timeout for a systemd unit. It is 15 sec by default. It don't surprises me that the containers are not able to start in that time. I would set to for the container services. Maybe make it configurable like other NixOS services? I will test that and create a PR if there are no objections.
I tried , but it still fails. It works with ! So it should be configurable, when even a very small container needs such a big value. But i'm not sure if this is a reliable solution. The load was up to 216 at boot! But the system was started in 6 minutes, with the workaround it took 27 minutes!
I wonder if setting on the units will help. This will give the units less cpu shares during boot, which might make the load go down a bit... From
I , but the load was not lower. It got up to 262!
I have one more idea. and I think this is also what the systemd people will tell you as an elegant solution to this problem. nspawn-containers can be socket-activated. So you create socket units, and the corresponding containers will be started on demand as soon as they receive traffic. this should spread out the startup a bit automatically if you're lucky and dont hit all the containers at the same time. () of course, the service inside has to support socket activation too for this to work. which doesn't. but there is a documented work around for that too: A full example is here, but
sounds like an interesting feature for us to implement. Maybe create another issue. For my usecase, it is better that the containers already run and don't have to start every time they are used.
That's not a requirement. Actually it seems more usual to use the scheme that the service is only started once &ndash; when the first connection comes.

Discussion:
RestartForceExitStatus = "133"; SuccessExitStatus = "133"; <ins> # Some containers take long to start # especially when you automatically start many at once TimeoutStartSec = cfg.timeoutStartSec; </ins> Restart = "on-failure"; Slice = "machine.slice";

extraVeths = {}; additionalCapabilities = []; ephemeral = false; <ins> timeoutStartSec = "15s"; </ins> allowedDevices = []; hostAddress = null; hostAddress6 = null;

''; }; <ins> timeoutStartSec = mkOption { type = types.str; default = "1min"; description = '' Time for the container to start. In case of a timeout, the container processes get killed. See <citerefentry><refentrytitle>systemd.time</refentrytitle> <manvolnum>7</manvolnum></citerefentry> for more information about the format. ''; }; </ins> bindMounts = mkOption { type = with types; loaOf (submodule bindMountOpts); default = {};