Repository: elixir-grpc/grpc
Pull request #112 (2019-08-30 12:47:35)

Add mechanism to set headers in Stub.connect/3
👋. I'm excited to see all the work and progress happening here with GRPC + Elixir! Thank you.

I spiked out a general solution for setting headers at the channel level using some of the existing tooling. I see this patch providing a general solution for https://github.com/elixir-grpc/grpc/pull/104 and https://github.com/elixir-grpc/grpc/pull/72, without needing a per header function set at the request level (better ux for users).

This patch provides a generic way to set headers that will be attached
to each request from this `channel` by re-using
append_custom_metadata/2.

The use case is to provide a location for setting authorization headers
that does not require manually adding it to each Stub fn call.

With this change, headers can be set as:

```
channel = Stub.connect(host, port, headers: [{"authorization", "Bearer TOKEN"}])
```

On the receiving server side they can be accessed using existing getter
function. For the authorization example, this can be done in an
Interceptor:

```elixir
defmodule Authenticator do
  def init(_), do: []
  def call(req, stream, next, opts) do
    with headers when is_map(headers) <- GRPC.Stream.get_headers(stream),
         token when not is_nil(token) <- Map.get(headers, "authorization") do
         .... # use token
         .... # if authenticated
         next.(req, stream)
    end
  end
end
```

Adds two tests, one to verify on client side that headers can be set and
accessible, a second test is for integration to verify the roundtrip of
a header from client to server, then encode the header into a GRPC
message response for test verification.

Thank you @clearjs for the initial concept that I used as initial concept leading to this PR.

@tony612 I'm happy to have feedback on approach or advice on extra areas to test that are unaddressed. 👍 

zph: @tony612 If I can put together a helper for regenerating `proto` in a more automatic fashion, would you like me to submit that in a separate PR? I like the idea of removing that as a manual step when in `:dev` so that proto and pb.ex never diverge.
clearjs: @zph thank you. It looks like you had more requirements in mind than I did in https://github.com/clearjs/grpc-elixir/commit/3dfa4648711ec0a59bb3bba2766de9ce770a0789. I only needed to be able to authenticate to Google PubSub using this application. I didn't spend enough time to understand your requirements and the approach you took, but I think that sharing mine might be useful.

So, with the above commit I'm able to write the following code:

```elixir
project_id="..." # your project_id from Google PubSub
token="..." # token string from Goth
{:ok, channel} = GRPC.Stub.connect("pubsub.googleapis.com:443", [cred: GRPC.Credential.new(ssl: [])])
req = Google.Pubsub.V1.ListTopicsRequest.new(project: "projects/#{project_id}", page_size: 5)
{_, _reply} = channel |> Google.Pubsub.V1.Publisher.Stub.list_topics(req, token: token, content_type: "application/grpc")
```

This way, I get authenticated to Google PubSub and can use it. I retrieve `token` using Goth configured with a secret json file generated to provide access to PubSub. I retrieve and provide this token each time I make an API call, so it's always fresh. I think that Goth refreshes it about once an hour and keeps the result cached. The downside is that it isn't convenient to pass it with each call.

So, my intention was instead to provide a function when creating the channel which would get the fresh value of token from Goth and return it. Something like:

```elixir
scope = "https://www.googleapis.com/auth/pubsub"
{:ok, channel} = GRPC.Stub.connect("pubsub.googleapis.com:443", [cred: GRPC.Credential.new(ssl: []), auth: fn () -> "Bearer #{Goth.Token.for_scope(scope).token}" end])
```

Then the grpc application should invoke this lambda function for each api request and add the result to authentication header just like I did above.

My understanding, after I spent very little time reading the code and text (it's late here in Ukraine), is that your approach is probably more generic and covers additional use cases, but it may after about an hour keep stale token value specified initially. Hope that this information helps.

Also, note that it is probably not needed to specify `content_type` explicitly anymore. I didn't check, but I remember somebody wrote that `"application/grpc"` is now the default value.
clearjs: @zph thank you. It looks like you had more requirements in mind than I did in https://github.com/clearjs/grpc-elixir/commit/3dfa4648711ec0a59bb3bba2766de9ce770a0789. I only needed to be able to authenticate to Google PubSub using this application. I didn't spend enough time to understand your requirements and the approach you took, but I think that sharing mine might be useful.

So, with the above commit I'm able to write the following code:

```elixir
project_id="..." # your project_id from Google PubSub
token="..." # token string from Goth
{:ok, channel} = GRPC.Stub.connect("pubsub.googleapis.com:443", [cred: GRPC.Credential.new(ssl: [])])
req = Google.Pubsub.V1.ListTopicsRequest.new(project: "projects/#{project_id}", page_size: 5)
{_, _reply} = channel |> Google.Pubsub.V1.Publisher.Stub.list_topics(req, token: token, content_type: "application/grpc")
```

This way, I get authenticated to Google PubSub and can use it. I retrieve `token` using Goth configured with a secret json file generated to provide access to PubSub. I retrieve and provide this token each time I make an API call, so it's always fresh. I think that Goth refreshes it about once an hour and keeps the result cached. The downside is that it isn't convenient to pass it with each call.

So, my intention was instead to provide a function when creating the channel which would get the fresh value of token from Goth and return it. Something like:

```elixir
scope = "https://www.googleapis.com/auth/pubsub"
{:ok, channel} = GRPC.Stub.connect("pubsub.googleapis.com:443", [
    cred: GRPC.Credential.new(ssl: []), 
    auth: fn () -> "Bearer #{Goth.Token.for_scope(scope).token}" end
])
```

Then the grpc application should invoke this lambda function for each api request and add the result as the `authorization` header just like I did above.

My understanding, after I spent very little time reading your code and comments (it's late here in Ukraine), is that your approach is probably more generic and covers additional use cases, but it may after about an hour keep stale token value specified initially. Hope that this information helps.

Also, note that it is probably not needed to specify `content_type` explicitly anymore. I didn't check, but I remember somebody wrote that `"application/grpc"` is now the default value.
clearjs: @zph thank you. Your approach is more generic than mine was in https://github.com/clearjs/grpc-elixir/commit/3dfa4648711ec0a59bb3bba2766de9ce770a0789. I only needed to be able to authenticate to Google Cloud (for example, PubSub). You allow specifying arbitrary headers per channel. Whether there are use cases for this is unclear, but this generalization might be very useful. It would help if a few possible use cases were mentioned.

So, with the above commit I'm able to write the following code:

```elixir
project_id="..." # your project_id from Google PubSub
token="..." # token string from Goth
{:ok, channel} = GRPC.Stub.connect("pubsub.googleapis.com:443", [cred: GRPC.Credential.new(ssl: [])])
req = Google.Pubsub.V1.ListTopicsRequest.new(project: "projects/#{project_id}", page_size: 5)
{_, _reply} = channel |> Google.Pubsub.V1.Publisher.Stub.list_topics(req, token: token, content_type: "application/grpc")
```

(Note that it is probably not necessary to specify `content_type` explicitly anymore. I didn't check, but I remember somebody wrote that `"application/grpc"` is now the default value.)

With the above code, I get authenticated to Google PubSub and can use it. I retrieve `token` using Goth configured with a secret JSON file generated to provide access to PubSub. I retrieve and provide this token each time I make an API call, so it's always fresh. I think that Goth refreshes it about once an hour and keeps the result cached. The downside is that it isn't convenient to pass it with each call.

I planned to allow passing a function when creating the channel. This function would get a fresh value of token from Goth and return it. Something like:

```elixir
scope = "https://www.googleapis.com/auth/pubsub"
{:ok, channel} = GRPC.Stub.connect("pubsub.googleapis.com:443", [
    cred: GRPC.Credential.new(ssl: []), 
    auth: fn () -> "Bearer #{Goth.Token.for_scope(scope).token}" end
])
```

Then the grpc application would invoke this lambda function for each api request and add the result as the `authorization` header just like I did above.

With your approach after some time (about an hour according to my tests) the channel may need to be recreated, because the token value specified initially would become stale. This can be solved in a generic way if you allow passing a function that would return a list of headers:


```elixir
scope = "https://www.googleapis.com/auth/pubsub"
{:ok, channel} = GRPC.Stub.connect("pubsub.googleapis.com:443", [
    cred: GRPC.Credential.new(ssl: []), 
    per_call_headers: fn () ->
      [{"authorization", "Bearer #{Goth.Token.for_scope(scope).token}"}]
    end
])
```

Passing constant headers can also be kept to simplify client code in cases where the token cannot change during the channel lifetime. But I'm not sure that the minor inconvenience of wrapping headers inside a function in, likely, just one place where the channel is created, is significant.

Patch:
diff --git a/lib/grpc/channel.ex b/lib/grpc/channel.ex
index 94b36ca..09194cc 100644
--- a/lib/grpc/channel.ex
+++ b/lib/grpc/channel.ex
@@ -26,7 +26,8 @@ defmodule GRPC.Channel do
           codec: module,
           interceptors: [],
           compressor: module,
-          accepted_compressors: [module]
+          accepted_compressors: [module],
+          headers: list
         }
   defstruct host: nil,
             port: nil,
@@ -37,5 +38,6 @@ defmodule GRPC.Channel do
             codec: GRPC.Codec.Proto,
             interceptors: [],
             compressor: nil,
-            accepted_compressors: []
+            accepted_compressors: [],
+            headers: []
 end
diff --git a/lib/grpc/stub.ex b/lib/grpc/stub.ex
index 2a83f05..b43ed7e 100644
--- a/lib/grpc/stub.ex
+++ b/lib/grpc/stub.ex
@@ -123,6 +123,7 @@ defmodule GRPC.Stub do
     * `:compressor` - the client will use this to compress requests and decompress responses. If this is set, accepted_compressors
         will be appended also, so this can be used safely without `:accesspted_compressors`.
     * `:accepted_compressors` - tell servers accepted compressors, this can be used without `:compressor`
+    * `:headers` - headers to attach to each request
   """
   @spec connect(String.t(), Keyword.t()) :: {:ok, GRPC.Channel.t()} | {:error, any}
   def connect(addr, opts \\ []) when is_binary(addr) and is_list(opts) do
@@ -155,6 +156,7 @@ defmodule GRPC.Stub do
     codec = Keyword.get(opts, :codec, GRPC.Codec.Proto)
     compressor = Keyword.get(opts, :compressor)
     accepted_compressors = Keyword.get(opts, :accepted_compressors) || []
+    headers = Keyword.get(opts, :headers) || []
 
     accepted_compressors =
       if compressor do
@@ -172,7 +174,8 @@ defmodule GRPC.Stub do
       interceptors: interceptors,
       codec: codec,
       compressor: compressor,
-      accepted_compressors: accepted_compressors
+      accepted_compressors: accepted_compressors,
+      headers: headers
     }
     |> adapter.connect(opts[:adapter_opts])
   end
@@ -185,13 +188,15 @@ defmodule GRPC.Stub do
         120_000
       end
 
-    jitter =
+    uniform_fn =
       if function_exported?(:rand, :uniform_real, 0) do
-        (:rand.uniform_real() - 0.5) / 2.5
+        :uniform_real
       else
-        (:rand.uniform() - 0.5) / 2.5
+        :uniform
       end
 
+    jitter = (apply(:rand, uniform_fn, []) - 0.5) / 2.5
+
     round(timeout + jitter * timeout)
   end
 
diff --git a/lib/grpc/transport/http2.ex b/lib/grpc/transport/http2.ex
index 9de576f..82f6a0c 100644
--- a/lib/grpc/transport/http2.ex
+++ b/lib/grpc/transport/http2.ex
@@ -40,6 +40,7 @@ defmodule GRPC.Transport.HTTP2 do
     ]
     |> append_compressor(stream.compressor)
     |> append_accepted_compressors(stream.accepted_compressors)
+    |> append_custom_metadata(stream.channel.headers)
     |> append_encoding(opts[:grpc_encoding])
     |> append_timeout(opts[:timeout])
     |> append_custom_metadata(opts[:metadata])


Test patch:
diff --git a/test/grpc/channel_test.exs b/test/grpc/channel_test.exs
index 610b56a..098ea01 100644
--- a/test/grpc/channel_test.exs
+++ b/test/grpc/channel_test.exs
@@ -13,4 +13,10 @@ defmodule GRPC.ChannelTest do
     {:ok, channel} = GRPC.Stub.connect("10.1.0.0:50051", adapter: ClientAdapter, cred: cred)
     assert %Channel{host: "10.1.0.0", port: 50051, scheme: "https", cred: ^cred} = channel
   end
+
+  test "connect/2 allows setting default headers" do
+    headers = [{"authorization", "Bearer TOKEN"}]
+    {:ok, channel} = GRPC.Stub.connect("10.1.0.0:50051", adapter: ClientAdapter, headers: headers)
+    assert %Channel{host: "10.1.0.0", port: 50051, headers: ^headers} = channel
+  end
 end
diff --git a/test/grpc/integration/server_test.exs b/test/grpc/integration/server_test.exs
index 559a804..6baafc6 100644
--- a/test/grpc/integration/server_test.exs
+++ b/test/grpc/integration/server_test.exs
@@ -20,6 +20,11 @@ defmodule GRPC.Integration.ServerTest do
     def say_hello(req, _stream) do
       Helloworld.HelloReply.new(message: "Hello, #{req.name}")
     end
+
+    def check_headers(_req, stream) do
+      token = GRPC.Stream.get_headers(stream)["authorization"]
+      Helloworld.HeaderReply.new(authorization: token)
+    end
   end
 
   defmodule HelloErrorServer do
@@ -157,4 +162,20 @@ defmodule GRPC.Integration.ServerTest do
       end)
     end)
   end
+
+  test "headers set on channel are present in receiving server" do
+    run_server([HelloServer], fn port ->
+      token = "Bearer TOKEN"
+
+      {:ok, channel} =
+        GRPC.Stub.connect("localhost:#{port}",
+          headers: [{"authorization", token}]
+        )
+
+      {:ok, reply} =
+        channel |> Helloworld.Greeter.Stub.check_headers(Helloworld.HeaderRequest.new())
+
+      assert reply.authorization == token
+    end)
+  end
 end
diff --git a/test/support/helloworld.pb.ex b/test/support/helloworld.pb.ex
index 7c65811..f8758c3 100644
--- a/test/support/helloworld.pb.ex
+++ b/test/support/helloworld.pb.ex
@@ -1,31 +1,56 @@
 defmodule Helloworld.HelloRequest do
-  use Protobuf
+  @moduledoc false
+  use Protobuf, syntax: :proto3
 
   @type t :: %__MODULE__{
           name: String.t()
         }
   defstruct [:name]
 
-  field :name, 1, optional: true, type: :string
+  field :name, 1, type: :string
 end
 
 defmodule Helloworld.HelloReply do
-  use Protobuf
+  @moduledoc false
+  use Protobuf, syntax: :proto3
 
   @type t :: %__MODULE__{
           message: String.t()
         }
   defstruct [:message]
 
-  field :message, 1, optional: true, type: :string
+  field :message, 1, type: :string
+end
+
+defmodule Helloworld.HeaderRequest do
+  @moduledoc false
+  use Protobuf, syntax: :proto3
+
+  @type t :: %__MODULE__{}
+  defstruct []
+end
+
+defmodule Helloworld.HeaderReply do
+  @moduledoc false
+  use Protobuf, syntax: :proto3
+
+  @type t :: %__MODULE__{
+          authorization: String.t()
+        }
+  defstruct [:authorization]
+
+  field :authorization, 1, type: :string
 end
 
 defmodule Helloworld.Greeter.Service do
+  @moduledoc false
   use GRPC.Service, name: "helloworld.Greeter"
 
   rpc :SayHello, Helloworld.HelloRequest, Helloworld.HelloReply
+  rpc :CheckHeaders, Helloworld.HeaderRequest, Helloworld.HeaderReply
 end
 
 defmodule Helloworld.Greeter.Stub do
+  @moduledoc false
   use GRPC.Stub, service: Helloworld.Greeter.Service
 end
diff --git a/test/support/helloworld.proto b/test/support/helloworld.proto
index cb91e82..0436df6 100644
--- a/test/support/helloworld.proto
+++ b/test/support/helloworld.proto
@@ -6,6 +6,7 @@ package helloworld;
 service Greeter {
   // Sends a greeting
   rpc SayHello (HelloRequest) returns (HelloReply) {}
+  rpc CheckHeaders (HeaderRequest) returns (HeaderReply) {}
 }
 
 // The request message containing the user's name.
@@ -17,3 +18,10 @@ message HelloRequest {
 message HelloReply {
   string message = 1;
 }
+
+message HeaderRequest {
+}
+
+message HeaderReply {
+  string authorization = 1;
+}