Repository: basecamp/kamal (Ruby)

Issue:
[Kamal 2] Error: Permission denied (publickey) only for interactive commands
I managed (after a few hiccups) to successfully deploy my Rails app to an Ubuntu 24.04 server. The user Im using is ubuntu, and it works fine after I added it to the docker group. I can deploy and I can even run kamal commands remotely like:

```
kamal app exec 'ruby -v'
Get most recent version available as an image...
Launching command with version latest from new container...
  INFO [d39a7214] Running docker run --rm --network kamal --env SOLID_QUEUE_IN_PUMA="true" --env JOB_CONCURRENCY="2" --env DB_HOST="mydbhost" --env-file .kamal/apps/scraper_admin/env/roles/web.env --volume scraper_admin_storage:/rails/storage augustosamame/scraper_admin:latest ruby -v on 18.211.91.90
  INFO [d39a7214] Finished in 0.925 seconds with exit status 0 (successful).
App Host: 18.211.91.90
ruby 3.3.4 (2024-07-09 revision be1089c8ec) [x86_64-linux]
```

however, running any of the interactive commands: kamal console, kamal logs, fails with Permission denied (publickey). The SSH .pem file Im using has to be fine, otherwise it would not work with any commands.

```
kamal app exec --interactive --reuse "bin/rails console"
Get current version of running container...
  INFO [076e4960] Running /usr/bin/env sh -c 'docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=scraper_admin --filter label=role=web --filter status=running --filter status=restarting --filter ancestor=$(docker image ls --filter reference=augustosamame/scraper_admin:latest --format '\''{{.ID}}'\'') ; docker ps --latest --format '\''{{.Names}}'\'' --filter label=service=scraper_admin --filter label=role=web --filter status=running --filter status=restarting' | head -1 | while read line; do echo ${line#scraper_admin-web-}; done on 18.211.91.90
  INFO [076e4960] Finished in 1.591 seconds with exit status 0 (successful).
Launching interactive command with version b7c1aec9139c8084bf9ea67f32fc12283303ad23 via SSH from existing container on 18.211.91.90...
ubuntu@18.211.91.90: Permission denied (publickey).
```

These are the relevant parts of my deploy.yml

```
aliases:
  console: app exec --interactive --reuse "bin/rails console"
  shell: app exec --interactive --reuse "bash"
  logs: app logs -f
  dbc: app exec --interactive --reuse "bin/rails dbconsole"

# Use a different ssh user than root
ssh:
  keys: [ "/path/to/my/pem" ]
  user: ubuntu
  log_level: debug
```

In my Dockerfile, I switched from the default rails user to ubuntu:

```
RUN groupadd --system --gid 1000 ubuntu && \
  useradd ubuntu --uid 1000 --gid 1000 --create-home --shell /bin/bash
RUN chown -R ubuntu:ubuntu db log storage tmp
USER ubuntu
```

but it did not fix the issue.





Discussion:
I just hit the same issue a few minutes ago and trying to debug. If I'm able to fix it or identify the issue, I'll ping back
Ditto. Looks like something breaks when `--interactive` is passed in.
I am also having same issue
look here https://github.com/basecamp/kamal/issues/218#issuecomment-2083590982
This also seems related to using a custom key in your ssh section of the kamal configuration in my case.
I managed to workaround this by using the config option and specifying the key from there instead of specifying it in kamal config
```
$ cat ~/.ssh/config

IdentityFile /root/.ssh/id_rsa.pem
```
For me happens the same error running in kamal 2.2.2, not only for the interactive command. 

My hotfix is run `ssh-add /path/to/my/pem` before running kamal commands.

I added the key to the IdentityFile option for the host in my system ssh config and it solved the issue for now. Will be nice to have the commands use the keys from the deploy.yml. 



Patch:
diff --git a/lib/kamal/commands/base.rb b/lib/kamal/commands/base.rb
index c0eac91cc..6d3f71ec9 100644
--- a/lib/kamal/commands/base.rb
+++ b/lib/kamal/commands/base.rb
@@ -11,7 +11,7 @@ def initialize(config)
     end
 
     def run_over_ssh(*command, host:)
-      "ssh#{ssh_proxy_args} -t #{config.ssh.user}@#{host} -p #{config.ssh.port} '#{command.join(" ").gsub("'", "'\\\\''")}'"
+      "ssh#{ssh_proxy_args}#{ssh_keys_args} -t #{config.ssh.user}@#{host} -p #{config.ssh.port} '#{command.join(" ").gsub("'", "'\\\\''")}'"
     end
 
     def container_id_for(container_name:, only_running: false)
@@ -94,5 +94,15 @@ def ssh_proxy_args
           " -o ProxyCommand='#{config.ssh.proxy.command_line_template}'"
         end
       end
+
+      def ssh_keys_args
+        "#{ ssh_keys.join("") if ssh_keys}" + "#{" -o IdentitiesOnly=yes" if config.ssh&.keys_only}"
+      end
+
+      def ssh_keys
+        config.ssh.keys&.map do |key|
+          " -i #{key}"
+        end
+      end
   end
 end


Test patch:
diff --git a/test/commands/app_test.rb b/test/commands/app_test.rb
index 1fb59e8a0..aad31f8a7 100644
--- a/test/commands/app_test.rb
+++ b/test/commands/app_test.rb
@@ -315,6 +315,16 @@ class CommandsAppTest < ActiveSupport::TestCase
     assert_equal "ssh -J root@2.2.2.2 -t app@1.1.1.1 -p 22 'ls'", new_command.run_over_ssh("ls", host: "1.1.1.1")
   end
 
+  test "run over ssh with keys config" do
+    @config[:ssh] = { "keys" => [ "path_to_key.pem" ] }
+    assert_equal "ssh -i path_to_key.pem -t root@1.1.1.1 -p 22 'ls'", new_command.run_over_ssh("ls", host: "1.1.1.1")
+  end
+
+  test "run over ssh with keys config with keys_only" do
+    @config[:ssh] = { "keys" => [ "path_to_key.pem" ], "keys_only" => true }
+    assert_equal "ssh -i path_to_key.pem -o IdentitiesOnly=yes -t root@1.1.1.1 -p 22 'ls'", new_command.run_over_ssh("ls", host: "1.1.1.1")
+  end
+
   test "run over ssh with proxy_command" do
     @config[:ssh] = { "proxy_command" => "ssh -W %h:%p user@proxy-server" }
     assert_equal "ssh -o ProxyCommand='ssh -W %h:%p user@proxy-server' -t root@1.1.1.1 -p 22 'ls'", new_command.run_over_ssh("ls", host: "1.1.1.1")