Repository: moby/moby (language: go)

Issue:
Docker rootless and outgoing IPv6
### Description

When using IPv6 in rootless mode in containers to reach any IPv6 address, this does not work.

Running ping6 fails, curl too:
```
docker@pcserver2023:~> docker run -it --rm bash:4.4
bash-4.4# ping6 -c 3 www.heise.de
PING www.heise.de (2a02:2e0:3fe:1001:7777:772e:2:85): 56 data bytes

--- www.heise.de ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
bash-4.4# wget --no-check-certificate --spider https://[2a02:2e0:3fe:1001:7777:772e:2:85]/index.html
Connecting to [2a02:2e0:3fe:1001:7777:772e:2:85] ([2a02:2e0:3fe:1001:7777:772e:2:85]:443)
wget: can't connect to remote host: Network unreachable
```

No matter what i have tried so far, i was not able to make outgoing ipv6 work (Think i have followed all instructions to make firewalls and docker work).
Strange but with slirp4netns at least connecting to my IPv6 from the internet (to my ipv6 adress of the hosts nic) is working perfectly.

Unfortunatelly my service is relying on the described functionality (Nextcloud AIO) - and i do only have IPv6 available.

Does anyone know any way to make outgoing IPv6 work on rootless?

btw. didn't get pasta, bypass4netns working, maybe some advice how to work around problems with slirp4netns and ipv6?

### Reproduce

1. fresh linux, enabled cgroupsv2. Set up Docker rootless as user docker.
Here using Network Pasta/implicit for testing, but its the same with slirp4netns:
```
docker@pcserver2023:~> curl -fsSL https://get.docker.com/rootless | sh
# Installing stable version 27.1.0
# Executing docker rootless install script, commit: 1ce4e39
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 70.4M  100 70.4M    0     0  2314k      0  0:00:31  0:00:31 --:--:-- 1052k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19.7M  100 19.7M    0     0  2292k      0  0:00:08  0:00:08 --:--:-- 2759k
+ PATH=/home/docker/bin:/home/docker/bin:/home/docker/bin:/home/docker/bin:/usr/local/bin:/usr/bin:/bin
+ /home/docker/bin/dockerd-rootless-setuptool.sh install
[INFO] Creating /home/docker/.config/systemd/user/docker.service
[INFO] starting systemd service docker.service
+ systemctl --user start docker.service
+ sleep 3
+ systemctl --user --no-pager --full status docker.service
● docker.service - Docker Application Container Engine (Rootless)
     Loaded: loaded (/home/docker/.config/systemd/user/docker.service; enabled; vendor preset: disabled)
    Drop-In: /home/docker/.config/systemd/user/docker.service.d
             └─override.conf
     Active: active (running) since Tue 2024-07-23 14:36:02 CEST; 3s ago
       Docs: https://docs.docker.com/go/rootless/
   Main PID: 934 (rootlesskit)
      Tasks: 45
     Memory: 67.1M
        CPU: 385ms
     CGroup: /user.slice/user-1001.slice/user@1001.service/app.slice/docker.service
             ├─ 934 rootlesskit --state-dir=/run/user/1001/dockerd-rootless --net=pasta --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=implicit --copy-up=/etc --copy-up=/run --propagation=rslave --ipv6 /home/docker/bin/dockerd-rootless.sh
             ├─ 940 /proc/self/exe --state-dir=/run/user/1001/dockerd-rootless --net=pasta --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=implicit --copy-up=/etc --copy-up=/run --propagation=rslave --ipv6 /home/docker/bin/dockerd-rootless.sh
             ├─ 955 pasta --foreground --stderr --ns-ifname=tap0 --mtu=1500 --no-dhcp --no-ra --address=10.0.2.100 --netmask=24 --gateway=10.0.2.2 --dns-forward=10.0.2.3 --no-map-gw --tcp-ports=auto --udp-ports=auto 940
             ├─ 967 dockerd
             └─ 992 containerd --config /run/user/1001/docker/containerd/containerd.toml
+ DOCKER_HOST=unix:///run/user/1001/docker.sock
+ /home/docker/bin/docker version
Client:
 Version:           27.1.0
 API version:       1.46
 Go version:        go1.21.12
 Git commit:        6312585
 Built:             Fri Jul 19 17:41:56 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.1.0
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.12
  Git commit:       a21b1a2
  Built:            Fri Jul 19 17:43:33 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.2
  ApiVersion:       1.1.1
  NetworkDriver:    pasta
  StateDir:         /run/user/1001/dockerd-rootless
+ systemctl --user enable docker.service
[INFO] Installed docker.service successfully.
[INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service`
[INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger docker`

[INFO] CLI context "rootless" already exists
[INFO] Using CLI context "rootless"
Current context is now "rootless"
Warning: DOCKER_HOST environment variable overrides the active context. To use "rootless", either set the global --context flag, or unset DOCKER_HOST environment variable.

[INFO] Make sure the following environment variable(s) are set (or add them to ~/.bashrc):
export PATH=/home/docker/bin:$PATH

[INFO] Some applications may require the following environment variable too:
export DOCKER_HOST=unix:///run/user/1001/docker.sock
```
daemon.json:
```
{
 "experimental": true,
 "ipv6": true,
 "fixed-cidr-v6": "fdff:6785:1::/48",
 "iptables": true,
 "ip6tables": true,
 "log-opts": {
  "max-size": "10m",
  "max-file": "5"
 }
}
```

network:
```
docker@pcserver2023:~> docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "61278e5b24b18a0b6897834cdad253851e5133389928e82a9fe1ebb9debda930",
        "Created": "2024-07-23T14:45:08.276573789+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": true,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                },
                {
                    "Subnet": "fdff:6785:1::/48",
                    "Gateway": "fdff:6785:1::1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]
```

### Expected behavior

should work as ipv4 does
docker run -it --rm bash:4.4
```
bash-4.4# ping -c 3 www.heise.de
PING www.heise.de (193.99.144.85): 56 data bytes
64 bytes from 193.99.144.85: seq=0 ttl=254 time=326.905 ms
64 bytes from 193.99.144.85: seq=1 ttl=254 time=14.479 ms
64 bytes from 193.99.144.85: seq=2 ttl=254 time=14.441 ms

--- www.heise.de ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 14.441/118.608/326.905 ms
```

### docker version

```bash
Client:
 Version:           27.1.0
 API version:       1.46
 Go version:        go1.21.12
 Git commit:        6312585
 Built:             Fri Jul 19 17:41:56 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.1.0
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.12
  Git commit:       a21b1a2
  Built:            Fri Jul 19 17:43:33 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          2.0.2
  ApiVersion:       1.1.1
  NetworkDriver:    pasta
  StateDir:         /run/user/1001/dockerd-rootless
```


### docker info

```bash
Client:
 Version:    27.1.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  0.16.2
    Path:     /usr/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /home/docker/.docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 35
 Server Version: 27.1.0
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 5.14.21-150500.55.68-default
 Operating System: openSUSE Leap 15.5
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 30.79GiB
 Name: pcserver2023
 ID: 45699224-ea9c-4865-8dea-a53bb20b788c
 Docker Root Dir: /home/docker/.local/share/docker
 Debug Mode: false
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
```


### Additional Info

Same setup with rootful works as expected, but has very different network devices (docker0), which are not existent on the host in rootless.


Discussion / hints:
What version of pasta (`pasta --version`) are you using? When it's up and running, how does pasta's command line look like (`ps aux|grep pasta`)?
I'm having the exact same problem as OP. In addition, it seems like publishing a port from rootless-docker only works for IPv4 and not IPv6. Running the exact same compose file from rootfull-docker, everything works fine,.

I've tried slirp4netns and pasta (RPM and compiled version). This is the pasta command:
```sh
docker@host:~$ ps -ef | grep -i pasta | grep -v grep
docker    158739     933  0 00:34 ?        00:00:00 rootlesskit --state-dir=/run/user/1004/dockerd-rootless --net=pasta --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=implicit --copy-up=/etc --copy-up=/run --propagation=rslave --ipv6 /usr/bin/dockerd-rootless.sh
docker    158746  158739  0 00:34 ?        00:00:00 /proc/self/exe --state-dir=/run/user/1004/dockerd-rootless --net=pasta --mtu=1500 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=implicit --copy-up=/etc --copy-up=/run --propagation=rslave --ipv6 /usr/bin/dockerd-rootless.sh
docker    158768  158739  1 00:34 ?        00:00:04 pasta --foreground --stderr --ns-ifname=tap0 --mtu=1500 --no-dhcp --no-ra --address=10.0.2.100 --netmask=24 --gateway=10.0.2.2 --dns-forward=10.0.2.3 --no-map-gw --tcp-ports=auto --udp-ports=auto 158746
```

I'm setting the following options for the docker service:
```sh
docker@host:~$ cat .config/systemd/user/docker.service.d/override.conf
[Service]
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_FLAGS=--ipv6"
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_NET=pasta"
Environment="DOCKERD_ROOTLESS_ROOTLESSKIT_PORT_DRIVER=implicit"
```

The network is created by the compose file:
```sh
docker@host:~/ipv6_test$ cat compose.yaml
services:
  test:
    image: jonlabelle/network-tools
    restart: unless-stopped
    entrypoint:
      - sleep
      - infinity
    networks:
      - ipv6-test

networks:
  ipv6-test:
    enable_ipv6: true
```

```sh
docker@host:~/ipv6_test$ docker network inspect ipv6_test_ipv6-test
[
    {
        "Name": "ipv6_test_ipv6-test",
        "Id": "58d0328ffe621dbe167c1c345a1064df1c0656cf1ce0605eb2f6b1e5db710daf",
        "Created": "2024-08-11T00:44:31.017336304+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": true,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.21.0.0/16",
                    "Gateway": "172.21.0.1"
                },
                {
                    "Subnet": "fd6c:8e02:d50b::/64",
                    "Gateway": "fd6c:8e02:d50b::1/64"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "6c466498f673b55670de0781787c105ff9d7d6624417d5982769fd4b873b2a5b": {
                "Name": "ipv6_test-test-1",
                "EndpointID": "a363996f9aec2e4656d56dbb7a614be051dd6d87c8142794f2bcab4537b0dd98",
                "MacAddress": "02:42:ac:15:00:02",
                "IPv4Address": "172.21.0.2/16",
                "IPv6Address": "fd6c:8e02:d50b::2/64"
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "ipv6-test",
            "com.docker.compose.project": "ipv6_test",
            "com.docker.compose.version": "2.29.1"
        }
    }
]
```
`--no-ra`, together with the fact that pasta isn't getting any IPv6 route explicitly configured, should effectively disable IPv6 (unless things are configured externally by rootlesskit).

@AkihiroSuda, I missed this when I first had a look at https://github.com/rootless-containers/rootlesskit/blob/1cfc96d53094debca536a480593b215138f13a24/pkg/network/pasta/pasta.go#L113: why do you need to disable router advertisements (`--no-ra`)? Is there something in rootlesskit replacing pasta's `--config-net`?
@obel1x, @laugmanuel, by the way, I had a quick look at https://github.com/moby/moby/issues/37779.

If you want to debug the (rootless) networking part of it without real containers, you can use `pasta` stand-alone: `pasta --config-net` will give you a connected shell in the new network namespace, and `pasta -d --config-net` will tell you everything that's going on.
> @obel1x, @laugmanuel, by the way, I had a quick look at #37779.
> 
> If you want to debug the (rootless) networking part of it without real containers, you can use `pasta` stand-alone: `pasta --config-net` will give you a connected shell in the new network namespace, and `pasta -d --config-net` will tell you everything that's going on.

Using `pasta -d --config-net` results in the following error:
```sh
docker@host:~$ pasta -d --config-net
0.0024: Couldn't write to /proc/self/uid_map0.0024: : Operation not permitted
0.0024: Couldn't configure user mappings
0.0026: Failed to clone process with detached namespaces0.0026: : Operation not permitted
```

I'm just wondering why IPv4 is working as it should and IPv6 is not?
I looked around for a solution that fits my needs and landed on the following setup which seems to work just fine:
- haproxy as host service listening on IPv4 and IPv6 (ports 80/443)
  - using the send-proxy-v2 option for the backend (= proxy protocol flag in TCP segment header)
- traefik in docker-rootless listening only locally (127.0.0.1) on IPv4 (ports 8080/8443)
  - configured to parse proxy-protocol header in TCP segment

This way I can still see the IPv4/IPv6 address in the Traefik logs and do all the SSL stuff there. 

**Edit**: and yes, I know... The topic of this thread is outgoing IPv6 traffic. However, the same problem exists for incoming IPv6 traffic on exposed ports and I thought debugging outgoing traffic should be easier and therefore joined this thread. **I'm still interested in a full solution and I'm happy in helping fine one...** 
> @AkihiroSuda, I missed this when I first had a look at https://github.com/rootless-containers/rootlesskit/blob/1cfc96d53094debca536a480593b215138f13a24/pkg/network/pasta/pasta.go#L113: why do you need to disable router advertisements (`--no-ra`)? Is there something in rootlesskit replacing pasta's `--config-net`?

I don't remember, but I guess I wanted to disable everything that is not needed for the basic functionality.
Can you try removing `--no-ra` and see if it works?
> Using `pasta -d --config-net` results in the following error:
> 
> ```shell
> docker@host:~$ pasta -d --config-net
> 0.0024: Couldn't write to /proc/self/uid_map0.0024: : Operation not permitted
> 0.0024: Couldn't configure user mappings
> 0.0026: Failed to clone process with detached namespaces0.0026: : Operation not permitted
> ```

Weird. Do you have `/proc/sys/kernel/unprivileged_userns_clone` set to `0` perhaps? What distribution is this? You're running this _outside_ the container, right? You mentioned RPMs, but this looks like a Debian or Ubuntu prompt. I wonder if we're hitting some issue related to https://gitlab.com/apparmor/apparmor/-/wikis/unprivileged_userns_restriction on Ubuntu.

By the way, thanks for sharing this output, I just posted [a patch](https://archives.passt.top/passt-dev/20240812082458.2504291-1-sbrivio@redhat.com/) to fix the double timestamping on debug messages there.

> I'm just wondering why IPv4 is working as it should and IPv6 is not?

It's not related to the error above. I guess it's more likely to be related to the fact that `--no-ra` is given by rootlesskit on pasta's command line, so, as @AkihiroSuda suggested, if you could try rebuilding rootlesskit dropping `--no-ra`  from [these options](https://github.com/rootless-containers/rootlesskit/blob/1cfc96d53094debca536a480593b215138f13a24/pkg/network/pasta/pasta.go#L113) that would be great.
> > @AkihiroSuda, I missed this when I first had a look at https://github.com/rootless-containers/rootlesskit/blob/1cfc96d53094debca536a480593b215138f13a24/pkg/network/pasta/pasta.go#L113: why do you need to disable router advertisements (`--no-ra`)? Is there something in rootlesskit replacing pasta's `--config-net`?
> 
> I don't remember, but I guess I wanted to disable everything that is not needed for the basic functionality.

By the way, I expect `--mtu 1500` to kill the [throughput](https://passt.top/passt/about/#performance_1).

As for the rest of the options, I would have opted passing `--config-net`, and not passing IPv4 addresses manually. For reference, when Podman uses pasta directly, it passes, by default, something like this: `--config-net --dns-forward 169.254.0.1 -t none -u none -T none -U none --no-map-gw --quiet --netns /run/user/1000/netns/netns-32d2389c-33c2-6e39-f136-99796ee58b0b`. Now, `--dns-forward`, port options, `--netns`, and `--quiet` are specific to the Podman integration, and I don't expect them to apply to rootlesskit. For the rest, @AkihiroSuda, should I review/try out different options, or you already investigated enough?

> > Using `pasta -d --config-net` results in the following error:
> > ```shell
> > docker@host:~$ pasta -d --config-net
> > 0.0024: Couldn't write to /proc/self/uid_map0.0024: : Operation not permitted
> > 0.0024: Couldn't configure user mappings
> > 0.0026: Failed to clone process with detached namespaces0.0026: : Operation not permitted
> > ```
> 
> Weird. Do you have `/proc/sys/kernel/unprivileged_userns_clone` set to `0` perhaps? What distribution is this? You're running this _outside_ the container, right? You mentioned RPMs, but this looks like a Debian or Ubuntu prompt. I wonder if we're hitting some issue related to https://gitlab.com/apparmor/apparmor/-/wikis/unprivileged_userns_restriction on Ubuntu.

I'm indeed using Ubuntu 24.04 LTS. `/proc/sys/kernel/unprivileged_userns_clone` is set to `1` (seems to be the default?).
But you were right with AppArmor - it's showing the following in `dmesg`:
```sh
root@host:~# dmesg -T | tail -1
[Mon Aug 12 23:20:09 2024] audit: type=1400 audit(1723497609.975:134): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=536796 comm="pasta.avx2" requested="userns_create" target="unprivileged_userns"
```

After setting `kernel.apparmor_restrict_unprivileged_userns` to `0`, the pasta command no longer throws errors and it enters the namespaced environment. And to my surprise, the following commands just work fine (don't worry about the dns stuff - this is because Ubuntu uses systemd-resolved by default, which is not available in the namespace):

```sh
root@host:~# curl --resolve google.com:443:172.217.18.110 https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>

root@host:~# curl --resolve google.com:443:[2a00:1450:4001:82a::200e] -6 https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>
```

Also, using the CLI arguments passed by docker-rootless __and__ adding `--config-net` also works fine:
```sh
pasta --config-net --foreground --stderr --ns-ifname=tap0 --mtu=1500 --no-dhcp --no-ra --address=10.0.2.100 --netmask=24 --gateway=10.0.2.2 --dns-forward=10.0.2.3 --no-map-gw --tcp-ports=auto --udp-ports=auto
```

> It's not related to the error above. I guess it's more likely to be related to the fact that `--no-ra` is given by rootlesskit on pasta's command line, so, as @AkihiroSuda suggested, if you could try rebuilding rootlesskit dropping `--no-ra` from [these options](https://github.com/rootless-containers/rootlesskit/blob/1cfc96d53094debca536a480593b215138f13a24/pkg/network/pasta/pasta.go#L113) that would be great.

I've compiled rootlesskit without the `--no-ra` option for pasta and restarted docker. I can see that `pasta` is now running without the option, but IPv6 egress is still not working.

Additionally, I've tried the following things without success:
- using the latest rootlesskit version (compiled from master) - same problem
- using the latest pasta version (compiled from master) - same problem
- using the default options set by docker-rootless __with__ `--config-net`  - causes pasta to fail initializing with `Couldn't set IPv4 address(es) in namespace: File exists`
> > > Using `pasta -d --config-net` results in the following error:
> > > ```shell
> > > docker@host:~$ pasta -d --config-net
> > > 0.0024: Couldn't write to /proc/self/uid_map0.0024: : Operation not permitted
> > > 0.0024: Couldn't configure user mappings
> > > 0.0026: Failed to clone process with detached namespaces0.0026: : Operation not permitted
> > > ```
> > 
> > Weird. Do you have `/proc/sys/kernel/unprivileged_userns_clone` set to `0` perhaps? What distribution is this? You're running this _outside_ the container, right? You mentioned RPMs, but this looks like a Debian or Ubuntu prompt. I wonder if we're hitting some issue related to https://gitlab.com/apparmor/apparmor/-/wikis/unprivileged_userns_restriction on Ubuntu.
> 
> I'm indeed using Ubuntu 24.04 LTS. `/proc/sys/kernel/unprivileged_userns_clone` is set to `1` (seems to be the default?). 

Right, that's the default.

> But you were right with AppArmor - it's showing the following in `dmesg`:
> 
> ```shell
> root@host:~# dmesg -T | tail -1
> [Mon Aug 12 23:20:09 2024] audit: type=1400 audit(1723497609.975:134): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=536796 comm="pasta.avx2" requested="userns_create" target="unprivileged_userns"
> ```
> 
> After setting `kernel.apparmor_restrict_unprivileged_userns` to `0`, the pasta command no longer throws errors and it enters the namespaced environment.

I need to find a workaround for that (it looks like default policies were changed on Ubuntu in a breaking way). I was waiting for an answer on https://github.com/kubevirt/kubevirt/issues/12333#issuecomment-2228074959 (same issue, it seems) but... whatever, I guess I'll have to check things on multiple distributions.

> And to my surprise, the following commands just work fine (don't worry about the dns stuff - this is because Ubuntu uses systemd-resolved by default, which is not available in the namespace):

By the way, pasta should play along with systemd-resolved nicely, see https://passt.top/passt/commit/?id=bad252687271c0255f6a077f19cbc19aa0427f8d.

> 
> ```shell
> root@host:~# curl --resolve google.com:443:172.217.18.110 https://google.com
> <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
> <TITLE>301 Moved</TITLE></HEAD><BODY>
> <H1>301 Moved</H1>
> The document has moved
> <A HREF="https://www.google.com/">here</A>.
> </BODY></HTML>
> 
> root@host:~# curl --resolve google.com:443:[2a00:1450:4001:82a::200e] -6 https://google.com
> <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
> <TITLE>301 Moved</TITLE></HEAD><BODY>
> <H1>301 Moved</H1>
> The document has moved
> <A HREF="https://www.google.com/">here</A>.
> </BODY></HTML>
> ```
> 
> Also, using the CLI arguments passed by docker-rootless **and** adding `--config-net` also works fine:
> 
> ```shell
> pasta --config-net --foreground --stderr --ns-ifname=tap0 --mtu=1500 --no-dhcp --no-ra --address=10.0.2.100 --netmask=24 --gateway=10.0.2.2 --dns-forward=10.0.2.3 --no-map-gw --tcp-ports=auto --udp-ports=auto
> ```
> 
> > It's not related to the error above. I guess it's more likely to be related to the fact that `--no-ra` is given by rootlesskit on pasta's command line, so, as @AkihiroSuda suggested, if you could try rebuilding rootlesskit dropping `--no-ra` from [these options](https://github.com/rootless-containers/rootlesskit/blob/1cfc96d53094debca536a480593b215138f13a24/pkg/network/pasta/pasta.go#L113) that would be great.
> 
> I've compiled rootlesskit without the `--no-ra` option for pasta and restarted docker. I can see that `pasta` is now running without the option, but IPv6 egress is still not working.

Thanks for the test. So, if I understood correctly, it works with stand-alone pasta, just not with Docker, correct?

> Additionally, I've tried the following things without success:
> 
>     * using the latest rootlesskit version (compiled from master) - same problem
> 
>     * using the latest pasta version (compiled from master) - same problem
> 
>     * using the default options set by docker-rootless **with** `--config-net`  - causes pasta to fail initializing with `Couldn't set IPv4 address(es) in namespace: File exists`

I guess that's why `--config-net` wasn't enabled with rootlesskit/Docker: the IPv4 address is already configured (by rootlesskit itself?), so if pasta tries to re-do that, it clashes. I guess we should tolerate an `EEXIST` error from pasta, eventually.

On the other hand, without `--config-net`, nobody configures IPv6 routes it seems. What do `ip -6 address show` and `ip -6 route show` report, in the container?

Meanwhile, I guess you could try, as a workaround, to manually add address and route in the container (pasta doesn't care about internal addresses and routes, it will just send out whatever it sees): `ip address add fc00::1/64 dev tap0` and `ip route add default via fc00::2` should do the trick (those are private, unique local addresses, so they won't conflict with anything).
> By the way, pasta should play along with systemd-resolved nicely, see https://passt.top/passt/commit/?id=bad252687271c0255f6a077f19cbc19aa0427f8d.

It's showing the following as it's trying to reach the resolver on the host:
```sh
root@host:~# dig google.com
;; communications error to 127.0.0.53#53: connection refused
;; communications error to 127.0.0.53#53: connection refused
;; communications error to 127.0.0.53#53: connection refused

; <<>> DiG 9.18.28-0ubuntu0.24.04.1-Ubuntu <<>> google.com
;; global options: +cmd
;; no servers could be reached
```

> Thanks for the test. So, if I understood correctly, it works with stand-alone pasta, just not with Docker, correct?
> 

Yes, correct.

> On the other hand, without `--config-net`, nobody configures IPv6 routes it seems. What do `ip -6 address show` and `ip -6 route show` report, in the container?

```sh
docker@host:~$ docker compose exec <container> ip -6 address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever

docker@host:~$ docker compose exec traefik ip -6 route show
docker@host:~$ 
```

So, the only IPv6 address is [::1] (loopback) and no routes are present.

> Meanwhile, I guess you could try, as a workaround, to manually add address and route in the container (pasta doesn't care about internal addresses and routes, it will just send out whatever it sees): `ip address add fc00::1/64 dev tap0` and `ip route add default via fc00::2` should do the trick (those are private, unique local addresses, so they won't conflict with anything).

This results in the following error:
```sh
/ # ip address add fc00::1/64 dev eth0
ip: RTNETLINK answers: Operation not permitted
```

The command is ran within the docker container running inside of docker-rootless.
> I need to find a workaround for that (it looks like default policies were changed on Ubuntu in a breaking way)

Maybe related to the same discussion as;

- https://github.com/containerd/containerd/pull/10123#issuecomment-2075805533
- https://github.com/moby/moby/pull/47749#pullrequestreview-2036591372
> /proc/sys/kernel/unprivileged_userns_clone

You may want to set `/proc/sys/kernel/apparmor_restrict_unprivileged_userns` to 0 too, if you want to skip writing an AppArmor profile to allow userns. (See `/etc/apparmor.d/rootlesskit` for an example profile for `/usr/bin/rootlesskit`)
> > I need to find a workaround for that (it looks like default policies were changed on Ubuntu in a breaking way)
> 
> Maybe related to the same discussion as;
> 
>     * [Update apparmor to allow confined runc to kill containers containerd/containerd#10123 (comment)](https://github.com/containerd/containerd/pull/10123#issuecomment-2075805533)
> 
>     * [apparmor: Allow confined runc to kill containers #47749 (review)](https://github.com/moby/moby/pull/47749#pullrequestreview-2036591372)

Thanks for the pointers, not the same rules, but the same conceptual breakage. I feel a bit better knowing I'm not the only one struggling with this. :)

> > /proc/sys/kernel/unprivileged_userns_clone
> 
> You may want to set `/proc/sys/kernel/apparmor_restrict_unprivileged_userns` to 0 too

Yes, @laugmanuel already tried here: https://github.com/moby/moby/issues/48257#issuecomment-2284985100 -- that works around the issue.

> if you want to skip writing an AppArmor profile to allow userns. (See `/etc/apparmor.d/rootlesskit` for an example profile for `/usr/bin/rootlesskit`)

We already ship an AppArmor profile, I have to fix it, I'm just not sure how to deal with openSUSE and I was waiting for an answer on https://github.com/kubevirt/kubevirt/issues/12333#issuecomment-2228074959. I guess I'll just have to try out the changes on Debian, openSUSE and Ubuntu myself.

The IPv6 configuration issue is a bit more complicated than that, though: it looks like with Docker and rootlesskit, IPv4 addresses and routes are configured autonomously (without relying on pasta), so we can't pass  `--config-net`. But without `--config-net`, IPv6 addresses and routes are not configured.

I can change pasta to accept `EEXIST` as we try to add IPv4 addresses and routes that were already configured. I wonder if you would prefer to implement a different solution in rootlesskit, though.

By the way, @laugmanuel, if you want to try out if IPv6 (only) works, you could pass `--config-net` and `-6` (`--ipv4-only`). With those, pasta will disable IPv4, and add IPv6 addresses and routes (only). It's not a solution, of course -- we need to fix that in another way.
> Thanks for the pointers, not the same rules, but the same conceptual breakage. I feel a bit better knowing I'm not the only one struggling with this. :)

You're not! At least, and I'm not an AppArmor expert (far from!) I know multiple other people were confused as well, so I thought I'd link to those discussions in case they provide information that help you (and others). I think those may still need some follow-ups overall to see if a more sane approach is possible (other than each project trying to keep an unbounded list of possible binaries).

Either way; I don't want to diverge from the topic too much if not needed 🙈
@AkihiroSuda, I was trying to fix this (the IPv6 configuration part, not the AppArmor issue) in pasta itself, at least in the short term. To sum up, the issue here is that rootlesskit doesn't ask pasta to configure the network, but it doesn't configure IPv6 addresses and routes by itself.

Now, If I add `--config-net` to pasta, as @laugmanuel reported, I get:
```
DEBU[0000] [rootlesskit:child ] executing [ip route add default via 10.0.2.2 dev tap0] 
DEBU[0000] [rootlesskit:parent] network/pasta: Couldn't set IPv4 address(es) in namespace: File exists 
```
fine, I thought, I'll just ignore the `EEXIST` error in pasta. But if I do that, rootlesskit will exit:
```
DEBU[0000] [rootlesskit:child ] executing [ip addr add 10.0.2.100/24 dev tap0] 
Error: ipv4: Address already assigned.
[rootlesskit:child ] error: executing [[ip link set tap0 up] [ip link set dev tap0 mtu 65520] [ip addr add 10.0.2.100/24 dev tap0] [ip route add default via 10.0.2.2 dev tap0]]: exit status 2
DEBU[0000] [rootlesskit:parent] killing pasta           
DEBU[0000] [rootlesskit:parent] killed pasta: signal: killed 
[rootlesskit:parent] error: child exited: exit status 2
```

And I can't really work around this in pasta itself.

At this point, my plan would be to send a pull request for rootlesskit that adjusts the rootlesskit integration by letting pasta doing its business of configuring the interface it owns, that is:
- dropping the call to `parentutils.PrepareTap()` in `ConfigureNetwork()`, pkg/network/pasta/pasta.go, because pasta can do that setup by itself (and faster, as it doesn't need to invoke `ip(8)`)
- avoiding the `ConfigureNetworkChild()` call in `setupNet()`, pkg/child/child.go, if the pasta back-end is selected

Before I start with that: does that sound reasonable to you?
> At this point, my plan would be to send a pull request for rootlesskit that adjusts the rootlesskit integration by letting pasta doing its business of configuring the interface it owns, that is:

Thanks

> avoiding the ConfigureNetworkChild() call in setupNet(), pkg/child/child.go, if the pasta back-end is selected

Why? Isn't this function already NOP?
https://github.com/rootless-containers/rootlesskit/blob/v2.2.0/pkg/network/pasta/pasta.go#L194-L197
> > avoiding the ConfigureNetworkChild() call in setupNet(), pkg/child/child.go, if the pasta back-end is selected
> 
> Why? Isn't this function already NOP? https://github.com/rootless-containers/rootlesskit/blob/v2.2.0/pkg/network/pasta/pasta.go#L194-L197

Sorry, of course! I meant activateDev(), called just after that.
SGTM
### About the AppArmor issue

I finally had the chance to check this on Ubuntu 23.10, 24.04, a current snapshot of the upcoming 24.10, a current openSUSE Tumbleweed version, and a current Debian unstable (sid) installation.

The issue occurs on Ubuntu 23.10 (`passt-0.0~git20230627.289301b-1`) and 24.04 (`passt-0.0~git20240220.1e6f92b-1`) only (not on 24.10, not on openSUSE, not on Debian) because, together with the change outlined in [Ubuntu's SE045 specification](https://discourse.ubuntu.com/t/spec-unprivileged-user-namespace-restrictions-via-apparmor-in-ubuntu-23-10/37626) and AppArmor's [wiki](https://gitlab.com/apparmor/apparmor/-/wikis/unprivileged_userns_restriction), a Debian package [commit](https://salsa.debian.org/sbrivio/passt/-/commit/4a77ef55c34c579d4845aa2dfd003abf2195ea9b) is also missing from those versions.

That commit actually includes the AppArmor profile for `pasta(1)` in the package. The AppArmor ABI of the profile is `3.0`, so it doesn't contain an explicit `allow userns create`, but the mere fact that there's a profile with ABI 3.0 allows pasta to create its sandboxing user namespace.

Quoting from Ubuntu's SE045 specification, one step for that change should have been:

> identify all packages within the Ubuntu archive that make use of unprivileged user namespaces

but this was somehow missed, I guess (I'm the maintainer of the Debian package, but I didn't get any notification).

Now, while Ubuntu 24.10 and openSUSE Tumbleweed ship AppArmor packages with support for the `4.0` ABI, Debian unstable still ships 3.1.17, so, to keep things simple and still ship a single AppArmor profile (developed upstream), I won't update the profile to ABI 4.0 yet. Updating the profile wouldn't solve the issue anyway.

So, how do we solve this? We would need to backport that Debian commit to Ubuntu 24.04 (and possibly 23.10), but I can't seem to register a Launchpad account to even start the [process](https://wiki.ubuntu.com/UbuntuBackports#Procedure) (wrong email address? :smile: ). If somebody could do that, or at least **file an Ubuntu issue**, that would be great. Thanks.

> file an Ubuntu issue

done: https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2077158

Fix patch:
diff --git a/Dockerfile b/Dockerfile
index 0e73fcbb61727..2ec128c7a5010 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -356,7 +356,7 @@ FROM base AS rootlesskit-src
 WORKDIR /usr/src/rootlesskit
 RUN git init . && git remote add origin "https://github.com/rootless-containers/rootlesskit.git"
 # When updating, also update vendor.mod and hack/dockerfile/install/rootlesskit.installer accordingly.
-ARG ROOTLESSKIT_VERSION=v2.0.2
+ARG ROOTLESSKIT_VERSION=v2.3.1
 RUN git fetch -q --depth 1 origin "${ROOTLESSKIT_VERSION}" +refs/tags/*:refs/tags/* && git checkout -q FETCH_HEAD
 
 FROM base AS rootlesskit-build
diff --git a/hack/dockerfile/install/rootlesskit.installer b/hack/dockerfile/install/rootlesskit.installer
index 27a0cf662f475..f080e9270767f 100755
--- a/hack/dockerfile/install/rootlesskit.installer
+++ b/hack/dockerfile/install/rootlesskit.installer
@@ -1,7 +1,7 @@
 #!/bin/sh
 
 # When updating, also update vendor.mod and Dockerfile accordingly.
-: "${ROOTLESSKIT_VERSION:=v2.0.2}"
+: "${ROOTLESSKIT_VERSION:=v2.3.1}"
 
 install_rootlesskit() {
 	case "$1" in
diff --git a/vendor.mod b/vendor.mod
index 7ef350e914901..0d198d8372f38 100644
--- a/vendor.mod
+++ b/vendor.mod
@@ -87,7 +87,7 @@ require (
 	github.com/pelletier/go-toml v1.9.5
 	github.com/pkg/errors v0.9.1
 	github.com/prometheus/client_golang v1.17.0
-	github.com/rootless-containers/rootlesskit/v2 v2.0.2
+	github.com/rootless-containers/rootlesskit/v2 v2.3.1
 	github.com/sirupsen/logrus v1.9.3
 	github.com/spf13/cobra v1.8.1
 	github.com/spf13/pflag v1.0.5
@@ -148,7 +148,7 @@ require (
 	github.com/containerd/stargz-snapshotter/estargz v0.15.1 // indirect
 	github.com/containerd/ttrpc v1.2.5 // indirect
 	github.com/containernetworking/cni v1.2.2 // indirect
-	github.com/containernetworking/plugins v1.4.0 // indirect
+	github.com/containernetworking/plugins v1.5.1 // indirect
 	github.com/cyphar/filepath-securejoin v0.2.4 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
 	github.com/dimchansky/utfbom v1.1.1 // indirect
diff --git a/vendor.sum b/vendor.sum
index 6dac211e0ecaf..f4dd07528fc90 100644
--- a/vendor.sum
+++ b/vendor.sum
@@ -174,8 +174,8 @@ github.com/containerd/typeurl/v2 v2.2.0 h1:6NBDbQzr7I5LHgp34xAXYF5DOTQDn05X58lsP
 github.com/containerd/typeurl/v2 v2.2.0/go.mod h1:8XOOxnyatxSWuG8OfsZXVnAF4iZfedjS/8UHSPJnX4g=
 github.com/containernetworking/cni v1.2.2 h1:9IbP6KJQQxVKo4hhnm8r50YcVKrJbJu3Dqw+Rbt1vYk=
 github.com/containernetworking/cni v1.2.2/go.mod h1:DuLgF+aPd3DzcTQTtp/Nvl1Kim23oFKdm2okJzBQA5M=
-github.com/containernetworking/plugins v1.4.0 h1:+w22VPYgk7nQHw7KT92lsRmuToHvb7wwSv9iTbXzzic=
-github.com/containernetworking/plugins v1.4.0/go.mod h1:UYhcOyjefnrQvKvmmyEKsUA+M9Nfn7tqULPpH0Pkcj0=
+github.com/containernetworking/plugins v1.5.1 h1:T5ji+LPYjjgW0QM+KyrigZbLsZ8jaX+E5J/EcKOE4gQ=
+github.com/containernetworking/plugins v1.5.1/go.mod h1:MIQfgMayGuHYs0XdNudf31cLLAC+i242hNm6KuDGqCM=
 github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
 github.com/coreos/etcd v3.3.13+incompatible h1:8F3hqu9fGYLBifCmRCJsicFqDx/D68Rt3q1JMazcgBQ=
 github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
@@ -599,8 +599,8 @@ github.com/rogpeppe/fastuuid v1.2.0/go.mod h1:jVj6XXZzXRy/MSR5jhDC/2q6DgLz+nrA6L
 github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
 github.com/rogpeppe/go-internal v1.11.0 h1:cWPaGQEPrBb5/AsnsZesgZZ9yb1OQ+GOISoDNXVBh4M=
 github.com/rogpeppe/go-internal v1.11.0/go.mod h1:ddIwULY96R17DhadqLgMfk9H9tvdUzkipdSkR5nkCZA=
-github.com/rootless-containers/rootlesskit/v2 v2.0.2 h1:wztWcDYFlk+EVAUuPJwlNMFXZIk1G14T45lv47WWGuA=
-github.com/rootless-containers/rootlesskit/v2 v2.0.2/go.mod h1:hE+ztevrQxNi+tdZyPKumzDk7VKDAf0E4seOzlOyBsY=
+github.com/rootless-containers/rootlesskit/v2 v2.3.1 h1:wdYtdKxWFvVLby9ThMP6O6/v2q/GmOXbkRi+4m9nPW0=
+github.com/rootless-containers/rootlesskit/v2 v2.3.1/go.mod h1:tdtfS9ak4bGmwJRmcjsAzcHN5rJ3c5dB7yhSV10KTbk=
 github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
 github.com/ryanuber/columnize v0.0.0-20160712163229-9b3edd62028f/go.mod h1:sm1tb6uqfes/u+d4ooFouqFdy9/2g9QGwK3SQygK0Ts=
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 2c02b1e23382d..f08f5461e6936 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -402,7 +402,7 @@ github.com/containernetworking/cni/pkg/types/create
 github.com/containernetworking/cni/pkg/types/internal
 github.com/containernetworking/cni/pkg/utils
 github.com/containernetworking/cni/pkg/version
-# github.com/containernetworking/plugins v1.4.0
+# github.com/containernetworking/plugins v1.5.1
 ## explicit; go 1.20
 github.com/containernetworking/plugins/pkg/ns
 # github.com/coreos/go-systemd/v22 v22.5.0
@@ -1043,8 +1043,8 @@ github.com/prometheus/common/model
 github.com/prometheus/procfs
 github.com/prometheus/procfs/internal/fs
 github.com/prometheus/procfs/internal/util
-# github.com/rootless-containers/rootlesskit/v2 v2.0.2
-## explicit; go 1.19
+# github.com/rootless-containers/rootlesskit/v2 v2.3.1
+## explicit; go 1.21.0
 github.com/rootless-containers/rootlesskit/v2/pkg/api
 github.com/rootless-containers/rootlesskit/v2/pkg/api/client
 github.com/rootless-containers/rootlesskit/v2/pkg/httputil


Test patch: