Change:
Describe the bug After updating the channel, IPFS does not start anymore. It would be nice if a migration would be done automatically if needed. Or at least document the process in the release notes. To Reproduce Steps to reproduce the behavior: ... ... ... Expected behavior IPFS just working Notify maintainers Metadata system: host os: multi-user?: sandbox: version: channels(root): nixpkgs: Maintainer information:
When i run it again, nothing happens. So we might just put it in a systemd service that runs every time before IPFS starts. But it might break and lead to data loss. That would be very bad! It is advised to backup before doing this. So it has to be discussed if we do such dangerous operations on peoples data automatically or just document how to do it manually.
There are strong opinions both for automatic migrations and against. Some software will bake migrations right into their primary executable and not even give you a chance. Other software will not fail if you don't run a migration and silently introduce strange breaking behavior that can be difficult to diagnose. Some migrations can be awkward to run on NixOS. As you mentioned some migrations are not safe to run without human interaction. Some migration programs aren't idempotent. Some software can damage it's own state if upgraded and run before migrations are executed. And to top it all off, the NixOS module system doesn't give us the ability to easily prevent a service from starting the first time after an upgrade. In summary this topic is very complex and hasn't been discussed in any public forum in detail. It would be nice if NixOS had a good solution for this in general and promoted standard desired behavior to module authors.
I think we should run the migrations automatically. It isn't great to have your system stop working. They don't really recommend backups, they say to do them if you want to be extra safe. I think we should just rely on the user having their own backups anyways. We should probably have a flag to control this, but I think default-enabled would be fine.
ping because they have become increasingly interested in these types of topics lately. ping who wants to smash all things state. :smile:
What am I doing wrong? EDIT: Also worked for me with
I used
Is there least some sort of one-shot systemd job that we could create that users would run by hand instead of the above? How are e.g. postgresql migrations handled today?
AFAIK, on postgresql migrations are always handled by applications themselves. Each application defines it's own ExecPreStart to run it's own migration. I was thinking that we could have a separate one-shot systemd job, and an option to run migrations automatically. If the automated migrations option is checked, then ipfs service would the migration job, and if it's not checked, then the user can run manually that job. I'm having a look at running IPFS at the moment. I've got a in case anyone is interested.
You can definitely run fs-repo-migrations in a dependency or PreStart command. Are you referring to Postgres version migrations or application schema migrations? Applications migrating their own schema is definitely the common case but I have yet to see an application that will automatically migrate your postgres from 9.2 to 9.3. The IPFS case is somewhat in the middle because it is both the application and the database. But in this case I think it makes sense to act like the application and run the migrations on startup.
yes, I was referring to schema migration. I've never seen version migration either.
I went with the easy solution, just running the migrator on preStart. The repo migration itself said that those were safe Happy to take feedback.

Discussion:
opt = options.services.ipfs; ipfsFlags = toString ([ <del> (optionalString  cfg.autoMount                   "--mount") (optionalString  cfg.enableGC                    "--enable-gc") (optionalString (cfg.serviceFdlimit != null)     "--manage-fdlimit=false") (optionalString (cfg.defaultMode == "offline")   "--offline") </del> <ins> (optionalString cfg.autoMount "--mount") (optionalString cfg.enableGC "--enable-gc") (optionalString (cfg.serviceFdlimit != null) "--manage-fdlimit=false") (optionalString (cfg.defaultMode == "offline") "--offline") </ins> (optionalString (cfg.defaultMode == "norouting") "--routing=none") ] ++ cfg.extraFlags); splitMulitaddr = addrRaw: lib.tail (lib.splitString "/" addrRaw); <del> multiaddrToListenStream = addrRaw: let </del> <ins> multiaddrToListenStream = addrRaw: let </ins> addr = splitMulitaddr addrRaw; s = builtins.elemAt addr; <del> in if s 0 == "ip4" && s 2 == "tcp" then "${s 1}:${s 3}" </del> <ins> in if s 0 == "ip4" && s 2 == "tcp" then "${s 1}:${s 3}" </ins> else if s 0 == "ip6" && s 2 == "tcp" <del> then "[${s 1}]:${s 3}" </del> <ins> then "[${s 1}]:${s 3}" </ins> else if s 0 == "unix" <del> then "/${lib.concatStringsSep "/" (lib.tail addr)}" </del> <ins> then "/${lib.concatStringsSep "/" (lib.tail addr)}" </ins> else null; # not valid for listen stream, skip <del> multiaddrToListenDatagram = addrRaw: let </del> <ins> multiaddrToListenDatagram = addrRaw: let </ins> addr = splitMulitaddr addrRaw; s = builtins.elemAt addr; <del> in if s 0 == "ip4" && s 2 == "udp" then "${s 1}:${s 3}" </del> <ins> in if s 0 == "ip4" && s 2 == "udp" then "${s 1}:${s 3}" </ins> else if s 0 == "ip6" && s 2 == "udp" <del> then "[${s 1}]:${s 3}" </del> <ins> then "[${s 1}]:${s 3}" </ins> else null; # not valid for listen datagram, skip <del> in { </del> <ins> in { </ins> ###### interface

dataDir = mkOption { type = types.str; <del> default = if versionAtLeast config.system.stateVersion "17.09" then "/var/lib/ipfs" else "/var/lib/ipfs/.ipfs"; </del> <ins> default = if versionAtLeast config.system.stateVersion "17.09" then "/var/lib/ipfs" else "/var/lib/ipfs/.ipfs"; </ins> description = "The data dir for IPFS"; };

description = "Whether IPFS should try to mount /ipfs and /ipns at startup."; }; <ins> autoMigrate = mkOption { type = types.bool; default = true; description = "Whether IPFS should try to run the fs-repo-migration at startup."; }; </ins> ipfsMountDir = mkOption { type = types.str; default = "/ipfs";

These are applied last, so may override configuration set by other options in this module. Keep in mind that this configuration is stateful; i.e., unsetting anything in here does not reset the value to the default! ''; <del> default = {}; </del> <ins> default = { }; </ins> example = { Datastore.StorageMax = "100GB"; Discovery.MDNS.Enabled = false;

extraFlags = mkOption { type = types.listOf types.str; description = "Extra flags passed to the IPFS daemon"; <del> default = []; </del> <ins> default = [ ]; </ins> }; localDiscovery = mkOption {

type = types.nullOr types.int; default = null; description = "The fdlimit for the IPFS systemd unit or <literal>null</literal> to have the daemon attempt to manage it"; <del> example = 64*1024; </del> <ins> example = 64 * 1024; </ins> }; startWhenNeeded = mkOption {

ipfs --offline config Mounts.FuseAllowOther --json true ipfs --offline config Mounts.IPFS ${cfg.ipfsMountDir} ipfs --offline config Mounts.IPNS ${cfg.ipnsMountDir} <ins> '' + optionalString cfg.autoMigrate '' ${pkgs.ipfs-migrator}/bin/fs-repo-migrations -y </ins> '' + concatStringsSep "n" (collect <del> isString (mapAttrsRecursive (path: value: # Using heredoc below so that the value is never improperly quoted '' read value <<EOF ${builtins.toJSON value} EOF ipfs --offline config --json "${concatStringsSep "." path}" "$value" '') ({ Addresses.API = cfg.apiAddress; Addresses.Gateway = cfg.gatewayAddress; Addresses.Swarm = cfg.swarmAddress; } // cfg.extraConfig)) ); </del> <ins> isString (mapAttrsRecursive (path: value: # Using heredoc below so that the value is never improperly quoted '' read value <<EOF ${builtins.toJSON value} EOF ipfs --offline config --json "${concatStringsSep "." path}" "$value" '') ({ Addresses.API = cfg.apiAddress; Addresses.Gateway = cfg.gatewayAddress; Addresses.Swarm = cfg.swarmAddress; } // cfg.extraConfig)) ); </ins> serviceConfig = { <del> ExecStart = ["" "${cfg.package}/bin/ipfs daemon ${ipfsFlags}"]; </del> <ins> ExecStart = [ "" "${cfg.package}/bin/ipfs daemon ${ipfsFlags}" ]; </ins> User = cfg.user; Group = cfg.group; } // optionalAttrs (cfg.serviceFdlimit != null) { LimitNOFILE = cfg.serviceFdlimit; };

systemd.sockets.ipfs-gateway = { wantedBy = [ "sockets.target" ]; socketConfig = { <del> ListenStream = let </del> <ins> ListenStream = let </ins> fromCfg = multiaddrToListenStream cfg.gatewayAddress; <del> in [ "" ] ++ lib.optional (fromCfg != null) fromCfg; ListenDatagram = let </del> <ins> in [ "" ] ++ lib.optional (fromCfg != null) fromCfg; ListenDatagram = let </ins> fromCfg = multiaddrToListenDatagram cfg.gatewayAddress; <del> in [ "" ] ++ lib.optional (fromCfg != null) fromCfg; </del> <ins> in [ "" ] ++ lib.optional (fromCfg != null) fromCfg; </ins> }; };

wantedBy = [ "sockets.target" ]; # We also include "%t/ipfs.sock" because there is no way to put the "%t" # in the multiaddr. <del> socketConfig.ListenStream = let </del> <ins> socketConfig.ListenStream = let </ins> fromCfg = multiaddrToListenStream cfg.apiAddress; <del> in [ "" "%t/ipfs.sock" ] ++ lib.optional (fromCfg != null) fromCfg; </del> <ins> in [ "" "%t/ipfs.sock" ] ++ lib.optional (fromCfg != null) fromCfg; </ins> }; };