Package: ec2fleetd
EC2fleetdEC2fleetd is the home-brewed EC2 instance resource set up automation daemon. Its
goal is to to make it easier to run the applications that do not cope well in
the spot fleet.(c) 2024 David Timber <dxdt 𝕒𝕥 dev.snart.me>EC2fleetd is made to address the following issues:Maintaining the persistent application data by having "volume pools"Persisting distributed scientific computing(BOINC) dataMaking home-grade game servers tolerant to spot interruptions (ArmA,
Minecraft, Rust ...)^1HA'ing cryptocurrency nodesDNS record updates for single-instance fleets for monolithic applications such
as MTA and game serversThe usual flow would beAn instance is launched in the fleetThe instance boots up and starts EC2fleetdEC2fleetd attaches an existing volume containing application data already
populated by the previous instance. Or create a volume from the snapshot if
requiredEC2fleetd starts the applicationThe application starts servicing off the data in the volumeUpon interruption,EC2fleetd polls interruption warning notice from the IMDSIn the even of valid interruption warning notice, EC2fleetd runs the
configured commands. The commands should usually do ...Warn users/players on the serverSave data and stop the applicationUnmount the volumeEC2fleetd notifies and exits. The instance is left runningAt the time of termination, the instance gets shut down and terminatedEC2fleetd also offers notifications via SNS and SQS. The contents of the
notifications can be set so that it's useful in troubleshooting when something
goes wrong. This cannot be done using CloudWatch alone.What you have to do on top of thisCustom scripts for your use caseOr multiple Exec lines could do the trick!Volume PoolProvisioning the volume pool according to the capacity of the fleetMaintain versions of data volume snapshots to accelerate the spin up processFor Bitcoin nodes, it just makes sense to renew the snapshot every now and
then because the blockchain data grows incrementally by design(the "c" volume source)Make your own manager app that periodically reclaims
old volumes if necessaryINSTALLMake/modify the IAM role for the instance. Seedoc/perms.mdfor
required permissions.Advanced: use resource groups to further isolate the scope of EC2fleetd's
operation^3.In the fleet imageInstall the systemd units insystemd. Install the Python module.python-mpipinstall--upgradeec2fleetdThe configThere are two options.Specify the config through the user data of the launch templateShip it with the imageWithout--userdataoption, EC2fleetd treats the user data as the daemon
config. This will give you more flexibility when the image is set up for more
than one application. To use that way, enable the non-instanced Systemd
serviceec2fleetd.service.To ship the config files with the image, place the config files in/etc/ec2fleetdand enable the instanced Systemd serviceec2fleetd@.service.
For example, to enable/etc/ec2fleetd/my-app.jsonc, runsystemctl enable ec2fleetd@my-app.service.Config ExamplesGame ServerA Minecraft server on spot would look like this.{
  "$schema": "http://schema.si-magic.com/fleet-init/json/userdata/0",

  "hostname": "my-world.mc",

  "domains": {
    "my-world.mc": {
      "attach-volume": [
        {
          "device": "/dev/xvdf",
          "source": "x",
          "volume-id": "vol-NNNNNNNNNNNNNNNNN",
          "exec": [
            {
              "lines": [
                {
                  "argv": [ "/bin/mount", "/mnt/mc" ]
                }
              ]
            }
          ]
        }
      ],

      "update-route53": [
        {
          "hostedzone": "ZZZZZZZZZZZZZZZZZZZZZ",
          "name": "my-world.mc",
          "ttl": 300
        }
      ],

      "exec": [
        {
          "on": [ "started" ],
          "lines": [
            { "argv": [ "/bin/systemctl", "start", "minecraft.service" ] }
          ]
        },
        {
          "on": [ "interrupted" ],
          "lines": [
            /*
             * A script would be used to warn the users and stop
             * the server after a certain amount of time.
             */
            { "argv": [ "/path/to/interrupt/script" ] }
          ]
        }
      ],

      "notify": [
        {
          "backend": "aws-sns", // backend module name
          "options": { // backend-specific data to be passed to the module
            "region": "REGION",
            "topic": "TOPIC_ARN"
          }
        }
      ]
    }
  }
}CryptoCouple of Bitcoin nodes.{
  "$schema": "http://schema.si-magic.com/fleet-init/json/userdata/0",

  "hostname": "node-{ami-i}.btcn.example.com",

  "domains": {
    "btcn.example.com": {
      "attach-volume": [
        {
          "device": "/dev/xvdf",
          "source": "p",
          "pool-name": "btc-nodes",
          "exec": [
            {
              "lines": [
                {
                  "argv": [ "/bin/mount", "/mnt/btc" ]
                }
              ]
            }
          ]
        }
      ],

      // Naturally, there should be a CNAME RR pointing to these.
      "update-route53": [
        {
          "hostedzone": "ZZZZZZZZZZZZZZZZZZZZZ",
          "name": "node-{ami-i}.btcn.example.com",
          "ttl": 300
        }
      ],

      "exec": [
        {
          "on": [ "started" ],
          "lines": [
            { "argv": [ "/bin/systemctl", "start", "bitcoin.service" ] }
          ]
        },
        {
          "on": [ "interrupted" ],
          "lines": [
            { "argv": [ "/bin/systemctl", "stop", "bitcoin.service" ] },
            { "argv": [ "/bin/umount", "-l", "/mnt/btc" ] }
          ]
        }
      ],

      "notify": [
        {
          "backend": "aws-sns", // backend module name
          "options": { // backend-specific data to be passed to the module
            "region": "REGION",
            "topic": "TOPIC_ARN"
          }
        }
      ]
    }
  }
}DisclaimerThis project is GPL'd.For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software.In other words, it means that the developer(s) of EC2fleetd cannot be held
responsible for any cost incurred from misuse or software error.QuirksThe "c" attach-volume sourceThe "c" volume source is a bad idea in large scale. In the ideal world, the
volume pool has to be provisioned by the sysadmin so the cost from volume pools
is somewhat predictable. Imagine a situation where all 100 instances in the
fleet are creating a volume every time they get launched because you forgot to
add "p".update-route53When EC2fleetd exist, it does not delete the RR's it has created during the init
process. This is intentional becauseupdate-route53is made as a cheap hack.
The problem where the clients end up in the wrong house exists(I don't know the
technical term for this. Sorry). For improved security and reliability, consider
following options.Elastic Beanstalk in conjunction with Route 53(BEST!)Use load balancers in conjunction with Route 53TLS stackDo server verification using CASNIVirtual host: HTTP/1.1 "Host" header, SMTP HELO or the likesHMACOpenVPN tls-taTODOsMulti-domain inits are not run in parallelAWS requests are painfully slow, with round-trip times range from 100's ms.
Nothing can be done for this, but at least we can alleviate the problem by using
AWS API in parallel.This was considered in the design phase, but had to release EC2fleetd without
parallelism due to the limitations of Python's concurrency facilities. It's
totally possible to make EC2fleetd this way. It's just that doing inter-thread
or inter-process signalling with Python is pain in the bum. I might as well have
written EC2fleetd in C.What about Azure and Google?Other CSP's do offer spot instances. I started writing code with the fact in
mind, but had to take some shortcuts because I was running out of time.
Obviously, other CSP's have different approaches when it comes to resource
addressing, types, lifecycle and so on... so much more research is needed to
make an abstraction layer that can accommodate the big players in cloud
computing.JSON SchemaThe schema url is only a placeholder at the moment. There is definitely a need
for one here in the grand scheme of things. SeeQuirksfor more.LRU and MRU^2The current EC2fleetd implementation picks one volume from the pool based on the
instance index. Some use cases may benefit from LRU and MRU policies.Unit testsIf I get paid to finish it, maybe.NotesAll of this can be done using a user data scriptFor single-instance fleets? Yeah. But EC2fleetd could still save you some time
on Route 53 updates. Also, updating DNS records directly can save you some money
on ELB.EC2fleetd is reentrantMeaning that you can stop and start EC2fleetd service once it successfully sets
up the configured domains. The second run should make no changes to any AWS
resources(well, except for Route 53 RR's). You should make your exec scripts
reentrant as well, or specify event attribute or use macros.Most Linux commands are designed in this way anyways.mount: most drivers do nothing if already mountedsystemctl start|stop: do nothing if the unit is already started/stopped