Repository: spegel-org/spegel (go)

Issue:
Preventing scenario where we end up being unable to pull any images, including spegel itself
### Describe the problem to be solved

We had an edge case where a combination of:
- Weird leftover port configuration - something was already running on the host port and swallowed all requests
- Spegel initialization procedure, where we also mirror the ghcr.io repo.
- [containerd mirror issues with mirror timeouts
](https://github.com/containerd/containerd/issues/10624) - if a mirror just hangs during response, containerd never falls back

What happens in this scenario for new nodes is the spegel init container pulls from ghcr.io (or whatever upstream) happily and installs the containerd configuration to use the host port as a mirror. Then the spegel main container comes in, and this time the pull hits the above containerd issue, and we can never pull again. The only to unstick ourselves was to SSH into each node and modify containerd configs/fix the weird leftover port.

Repro:
- Use containerd on the k8s host node
- Put a blackhole on the host port of the spegel service so that any HTTP requests made just hang forever
- Install spegel and observe containerd failing to pull the image

### Proposed solution to the problem

I THINK the simplest way to prevent this is to mount the host port to the init container - if the port is already occupied by something else, we never initialize the configs. I'm not sure if this actually works in k8s land, but feels simple.

The next one is simply documentation that this could happen, and we don't mirror ghcr.io or some special repository just for spegel itself. Sort of janky, but works well enough for us.

Another way to prevent the worst case is to implement the uninstall hook mentioned here: https://github.com/spegel-org/spegel/issues/67. At least that way we can unstick ourselves and resolve the weird port issue. Not sure how that would work given all image pulls could be stuck though.

Pull request:
Add dial timeout configuration in Containerd mirror configuration
This change sets the dial timeout for Spegel at 200ms. The value is currently set as a static value for simplicity. This is not a timeout for the HTTP request but for how long Containerd should wait for a TCP connection. We can expect the dial time to be low as the networking is local. Setting this value helps reduce issues when Spegel is not running on a node which may cause a pull to get stuck indefinably. 

This feature was added in https://github.com/containerd/containerd/pull/11106 which means that Containerd version 2.1.0 or later has to be used for this configuration to have any effect. 

Fixes #739 

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 8e0076a..c2a3645 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Added
 
 - [#877](https://github.com/spegel-org/spegel/pull/877) Add support for www authenticate header.
+- [#878](https://github.com/spegel-org/spegel/pull/878) Add dial timeout configuration in Containerd mirror configuration.
 
 ### Changed
 
diff --git a/pkg/oci/containerd.go b/pkg/oci/containerd.go
index c01261a..f0b54b0 100644
--- a/pkg/oci/containerd.go
+++ b/pkg/oci/containerd.go
@@ -555,6 +555,7 @@ func templateHosts(mirroredRegistry url.URL, mirrorTargets []url.URL, capabiliti
 {{ range .MirrorTargets }}
 [host.'{{ .String }}']
 capabilities = {{ $.Capabilities }}
+dial_timeout = '200ms'
 {{- if $authorization }}
 [host.'{{ .String }}'.header]
 Authorization = '{{ $authorization }}'


Test patch:
diff --git a/pkg/oci/containerd_test.go b/pkg/oci/containerd_test.go
index 3aa1a9e..5e3b79c 100644
--- a/pkg/oci/containerd_test.go
+++ b/pkg/oci/containerd_test.go
@@ -333,12 +333,15 @@ func TestMirrorConfiguration(t *testing.T) {
 
 [host.'http://127.0.0.1:5000']
 capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'
 
 [host.'http://127.0.0.2:5000']
 capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'
 
 [host.'http://127.0.0.1:5001']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 			},
 		},
 		{
@@ -349,7 +352,8 @@ capabilities = ['pull', 'resolve']`,
 			prependExisting: false,
 			expectedFiles: map[string]string{
 				"_default/hosts.toml": `[host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 			},
 		},
 		{
@@ -362,11 +366,13 @@ capabilities = ['pull', 'resolve']`,
 				"docker.io/hosts.toml": `server = 'https://registry-1.docker.io'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull']`,
+capabilities = ['pull']
+dial_timeout = '200ms'`,
 				"foo.bar:5000/hosts.toml": `server = 'http://foo.bar:5000'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull']`,
+capabilities = ['pull']
+dial_timeout = '200ms'`,
 			},
 		},
 		{
@@ -380,11 +386,13 @@ capabilities = ['pull']`,
 				"docker.io/hosts.toml": `server = 'https://registry-1.docker.io'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 				"foo.bar:5000/hosts.toml": `server = 'http://foo.bar:5000'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 			},
 		},
 		{
@@ -398,11 +406,13 @@ capabilities = ['pull', 'resolve']`,
 				"docker.io/hosts.toml": `server = 'https://registry-1.docker.io'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 				"foo.bar:5000/hosts.toml": `server = 'http://foo.bar:5000'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 			},
 		},
 		{
@@ -422,11 +432,13 @@ capabilities = ['pull', 'resolve']`,
 				"docker.io/hosts.toml": `server = 'https://registry-1.docker.io'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 				"foo.bar:5000/hosts.toml": `server = 'http://foo.bar:5000'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 			},
 		},
 		{
@@ -448,11 +460,13 @@ capabilities = ['pull', 'resolve']`,
 				"docker.io/hosts.toml": `server = 'https://registry-1.docker.io'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 				"foo.bar:5000/hosts.toml": `server = 'http://foo.bar:5000'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 			},
 		},
 		{
@@ -495,6 +509,7 @@ client = ['/etc/certs/xxx/client.cert', '/etc/certs/xxx/client.key']`,
 
 [host.'http://127.0.0.1:5000']
 capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'
 
 [host.'http://example.com:30020']
 capabilities = ['pull', 'resolve']
@@ -510,7 +525,8 @@ client = ['/etc/certs/xxx/client.cert', '/etc/certs/xxx/client.key']`,
 				"foo.bar:5000/hosts.toml": `server = 'http://foo.bar:5000'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 			},
 		},
 		{
@@ -552,11 +568,13 @@ client = ['/etc/certs/xxx/client.cert', '/etc/certs/xxx/client.key']`,
 				"docker.io/hosts.toml": `server = 'https://registry-1.docker.io'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 				"foo.bar:5000/hosts.toml": `server = 'http://foo.bar:5000'
 
 [host.'http://127.0.0.1:5000']
-capabilities = ['pull', 'resolve']`,
+capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'`,
 			},
 		},
 		{
@@ -572,11 +590,13 @@ capabilities = ['pull', 'resolve']`,
 
 [host.'http://127.0.0.1:5000']
 capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'
 [host.'http://127.0.0.1:5000'.header]
 Authorization = 'Basic aGVsbG86d29ybGQ='
 
 [host.'http://127.0.0.1:5001']
 capabilities = ['pull', 'resolve']
+dial_timeout = '200ms'
 [host.'http://127.0.0.1:5001'.header]
 Authorization = 'Basic aGVsbG86d29ybGQ='`,
 			},