Repository: liftbridge-io/liftbridge
Pull request #130 (2019-11-03 10:43:50)

Enhancements for Liftbridge operator

annismckenzie: I dumped every change I needed for the Liftbridge operator to work correctly in here – let's talk about the pieces. I can move the health check probe thing to another MR or we can go with another solution entirely.
annismckenzie: May I ask whether you could add me as a collaborator on this project so that the tests can run successfully? 🙈 Or maybe there's a way to configure CircleCI to allow PRs to successfully run even when they need secrets from the main repository?
tylertreat: @annismckenzie Odd, so it seems CircleCI is not setting the environment variable for your pull requests? I would expect them to be set on all builds. Will take a look...
tylertreat: I think I've fixed the CI issue.
annismckenzie: > I think I've fixed the CI issue.

Yup, it's working now.
annismckenzie: I'll add a test so that the test coverage doesn't decrease. 🙈
annismckenzie: Tests are now failing because of the rename of the `liftbridge-grpc` module to `liftbridge-api`. I already renamed all the import paths but it still complains with:
```
github.com/liftbridge-io/liftbridge-api: module github.com/liftbridge-io/liftbridge-api@latest (v0.0.0-20190910222614-5694b15f251d) found, but does not contain package github.com/liftbridge-io/liftbridge-api
```

I mean, yes, that project doesn't have a go.mod file…
annismckenzie: > Tests are now failing because of the rename of the `liftbridge-grpc` module to `liftbridge-api`. I already renamed all the import paths but it still complains with:
> 
> ```
> github.com/liftbridge-io/liftbridge-api: module github.com/liftbridge-io/liftbridge-api@latest (v0.0.0-20190910222614-5694b15f251d) found, but does not contain package github.com/liftbridge-io/liftbridge-api
> ```
> 
> I mean, yes, that project doesn't have a go.mod file…

You'll have to fix all of the import paths in all of the projects until the tests will succeed again.
tylertreat: Oops, will get that fixed.
tylertreat: @annismckenzie This should be fixed now. Apologies for the breaking change!

Patch:
diff --git a/Makefile b/Makefile
index 48d2526..32760d5 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
 .PHONY: compose-up compose-down kind-up kind-down kind-apply kind-export
 
-NAME=kind
-KUBECONFIG:=~/.kube/kind-config-$(NAME)
+KIND_CLUSTER_NAME=kind
+KIND_KUBECONFIG:=~/.kube/kind-config-$(KIND_CLUSTER_NAME)
 
 compose-up:
 	@ cd dev/; docker-compose up --build
@@ -9,15 +9,21 @@ compose-up:
 compose-down:
 	@ cd dev/; docker-compose down --rmi local
 
+push-k8s-image:
+	@ skaffold run -p deploy-k8s-image
+
 kind-up:
-	kind create cluster --config k8s/dev/kind.yaml --name=$(NAME)
+	kind create cluster --config k8s/dev/kind.yaml --name=$(KIND_CLUSTER_NAME)
 
 kind-down:
-	kind delete cluster --name=$(NAME)
+	kind delete cluster --name=$(KIND_CLUSTER_NAME)
 
 kind-apply:
-	@ KUBECONFIG=$(KUBECONFIG) kubectl apply -R -f k8s/dev/nats-operator.yaml
-	@ KUBECONFIG=$(KUBECONFIG) skaffold dev -p dev
+	@ KUBECONFIG=$(KIND_KUBECONFIG) kubectl apply -R -f k8s/dev/nats-operator.yaml
+	@ KUBECONFIG=$(KIND_KUBECONFIG) skaffold dev -p dev
 
 kind-export:
-	@echo export KUBECONFIG="$$(kind get kubeconfig-path --name="$(NAME)")"
+	@echo export KUBECONFIG="$$(kind get kubeconfig-path --name="$(KIND_CLUSTER_NAME)")"
+
+build:
+	@ CGO_ENABLED=0 go build -mod=readonly -o liftbridge
diff --git a/documentation/client_implementation.md b/documentation/client_implementation.md
index 792579f..d1b8cd1 100644
--- a/documentation/client_implementation.md
+++ b/documentation/client_implementation.md
@@ -11,7 +11,7 @@ inaccurate, feel free to make a PR fixing it.
 
 Liftbridge relies on gRPC and Protocol Buffers (protobufs) for its API. These
 definitions are stored in the
-[liftbridge-grpc](https://github.com/liftbridge-io/liftbridge-grpc) repository.
+[liftbridge-grpc](https://github.com/liftbridge-io/liftbridge-api) repository.
 Refer to the [gRPC documentation](https://grpc.io/docs/) for guidance on
 generating the gRPC code from the protobuf definitions for your particular
 programming language. The liftbridge-grpc repository also contains generated
@@ -191,7 +191,7 @@ match the subject). Since Liftbridge streams attach to normal NATS subjects,
 it's also possible to [publish messages directly to NATS](https://github.com/liftbridge-io/go-liftbridge#publishing-directly-with-nats)
 using a [NATS client](https://nats.io/download/). Liftbridge works fine with
 plain, opaque NATS messages, but it also extends NATS with a [protobuf-based
-envelope protocol](https://github.com/liftbridge-io/liftbridge-grpc). This
+envelope protocol](https://github.com/liftbridge-io/liftbridge-api). This
 allows publishers to add metadata to messages like a key, headers, and acking
 information. Liftbridge client libraries may provide helper methods to make it
 easy to create envelopes and deal with acks yourself using a NATS client
@@ -269,7 +269,7 @@ func main() {
 	nc, _ := nats.Connect(nats.DefaultURL)
 
 	// Publish a message.
-	nc.Publish("foo.bar", []byte("Hello, world!")) 
+	nc.Publish("foo.bar", []byte("Hello, world!"))
 	nc.Flush()
 }
 ```
diff --git a/documentation/clients.md b/documentation/clients.md
index 03e0ff7..a53d5e2 100644
--- a/documentation/clients.md
+++ b/documentation/clients.md
@@ -6,7 +6,7 @@ title: Clients
 Currently, there are limited client libraries available for Liftbridge.
 However, Liftbridge uses gRPC for its client API, so low-level client libraries
 can be generated quite easily using the [Liftbridge protobuf
-definitions](https://github.com/liftbridge-io/liftbridge-grpc). See the [client
+definitions](https://github.com/liftbridge-io/liftbridge-api). See the [client
 implementation guidance](client_implementation.md) documentation for help on
 implementing a Liftbridge client library.
 
diff --git a/go.mod b/go.mod
index a5812f5..d443c57 100644
--- a/go.mod
+++ b/go.mod
@@ -4,39 +4,22 @@ go 1.13
 
 require (
 	github.com/Workiva/go-datastructures v1.0.50
-	github.com/armon/go-metrics v0.3.0 // indirect
 	github.com/dustin/go-humanize v1.0.0
 	github.com/golang/protobuf v1.3.2
 	github.com/hako/durafmt v0.0.0-20191009132224-3f39dc1ed9f4
-	github.com/hashicorp/go-hclog v0.10.0 // indirect
-	github.com/hashicorp/go-immutable-radix v1.1.0 // indirect
-	github.com/hashicorp/go-uuid v1.0.1 // indirect
-	github.com/hashicorp/golang-lru v0.5.3 // indirect
 	github.com/hashicorp/raft v1.1.1
 	github.com/hashicorp/raft-boltdb v0.0.0-20191021154308-4207f1bf0617
-	github.com/konsorten/go-windows-terminal-sequences v1.0.2 // indirect
 	github.com/liftbridge-io/go-liftbridge v0.0.0-20191106180712-4f7e4b9d8611
 	github.com/liftbridge-io/liftbridge-api v0.0.0-20190910222614-5694b15f251d
 	github.com/liftbridge-io/nats-on-a-log v0.0.0-20190703144237-760cefbfc85e
 	github.com/natefinch/atomic v0.0.0-20150920032501-a62ce929ffcc
-	github.com/nats-io/gnatsd v1.4.1 // indirect
 	github.com/nats-io/nats-server/v2 v2.1.0
 	github.com/nats-io/nats.go v1.9.1
-	github.com/nats-io/nkeys v0.1.2 // indirect
 	github.com/nats-io/nuid v1.0.1
 	github.com/nsip/gommap v0.0.0-20181229045655-f7881c3a959f
 	github.com/pkg/errors v0.8.1
 	github.com/sirupsen/logrus v1.4.2
 	github.com/stretchr/testify v1.4.0
 	github.com/urfave/cli v1.22.1
-	golang.org/x/crypto v0.0.0-20191105034135-c7e5f84aec59 // indirect
-	golang.org/x/net v0.0.0-20191105084925-a882066a44e0 // indirect
-	golang.org/x/sys v0.0.0-20191105231009-c1f44814a5cd // indirect
-	google.golang.org/genproto v0.0.0-20191028173616-919d9bdd9fe6 // indirect
 	google.golang.org/grpc v1.25.0
-	gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect
-	gopkg.in/yaml.v2 v2.2.5 // indirect
-	launchpad.net/gocheck v0.0.0-20140225173054-000000000087 // indirect
 )
-
-replace github.com/golang/lint => golang.org/x/lint v0.0.0-20190409202823-959b441ac422
diff --git a/go.sum b/go.sum
index 3694aea..56f8541 100644
--- a/go.sum
+++ b/go.sum
@@ -45,6 +45,7 @@ github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/me
 github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
+github.com/golang/lint v0.0.0-20180702182130-06c8688daad7/go.mod h1:tluoj9z5200jBnyusfRPU2LqT6J+DAorxEvtC7LHB+E=
 github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
diff --git a/k8s/dev/Dockerfile.kind b/k8s/Dockerfile.k8s
similarity index 62%
rename from k8s/dev/Dockerfile.kind
rename to k8s/Dockerfile.k8s
index dcf535e..85cfc3c 100644
--- a/k8s/dev/Dockerfile.kind
+++ b/k8s/Dockerfile.k8s
@@ -1,6 +1,6 @@
 FROM golang:1.13-alpine as build-base
 RUN apk update && apk upgrade && \
-    apk add --no-cache bash git openssh make bzr
+    apk add --no-cache bash git openssh make bzr wget
 WORKDIR /workspace
 ENV GO111MODULE on
 
@@ -18,8 +18,13 @@ COPY server/ server/
 # Build
 RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 GO111MODULE=on go build -mod=readonly -o liftbridge
 
+RUN GRPC_HEALTH_PROBE_VERSION=v0.3.1 && \
+    wget -qO /bin/grpc_health_probe https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/${GRPC_HEALTH_PROBE_VERSION}/grpc_health_probe-linux-amd64 && \
+    chmod +x /bin/grpc_health_probe
+
 FROM alpine:latest
 RUN apk update && apk add --no-cache bash
 COPY --from=build-base /workspace/liftbridge /usr/local/bin/liftbridge
-COPY k8s/dev/entrypoint.sh /entrypoint.sh
+COPY --from=build-base /bin/grpc_health_probe /bin/grpc_health_probe
+COPY k8s/entrypoint.sh /entrypoint.sh
 ENTRYPOINT ["/entrypoint.sh"]
diff --git a/k8s/base/config.yaml b/k8s/base/config.yaml
index 64078e1..052c054 100644
--- a/k8s/base/config.yaml
+++ b/k8s/base/config.yaml
@@ -4,6 +4,10 @@ data:
     listen: 0.0.0.0:9292
     log.level: debug
 
+    nats {
+      servers: ["nats://nats.liftbridge.svc:4222"]
+    }
+
     clustering {
       min.insync.replicas: 1
     }
diff --git a/k8s/base/statefulset.yaml b/k8s/base/statefulset.yaml
index a87f8fe..4cbac4a 100644
--- a/k8s/base/statefulset.yaml
+++ b/k8s/base/statefulset.yaml
@@ -21,21 +21,17 @@ spec:
           ports:
             - name: grpc
               containerPort: 9292
+          readinessProbe:
+            exec:
+              command: ["/bin/grpc_health_probe", "-service=proto.API", "-addr=:9292"]
+            initialDelaySeconds: 5
           volumeMounts:
             - name: liftbridge-data
               mountPath: /data
             - mountPath: /etc/liftbridge.conf
-              name: config
+              name: liftbridge-config
               subPath: liftbridge.conf
       volumes:
         - configMap:
             name: config
-          name: config
-  volumeClaimTemplates:
-    - metadata:
-        name: liftbridge-data
-      spec:
-        accessModes: ["ReadWriteOnce"]
-        resources:
-          requests:
-            storage: 5Gi
+          name: liftbridge-config
diff --git a/k8s/dev/entrypoint.sh b/k8s/dev/entrypoint.sh
deleted file mode 100755
index 08f508e..0000000
--- a/k8s/dev/entrypoint.sh
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/bin/bash
-
-if [ "$HOSTNAME" = "liftbridge-0" ]
-then
-  echo "Running in liftbridge-0 as a bootstrap seed..."
-  liftbridge --data-dir=/data --config /etc/liftbridge.conf --nats-servers=nats://nats.liftbridge.svc:4222 --raft-bootstrap-seed --id="$HOSTNAME"
-else
-  echo "Running in $HOSTNAME..."
-  liftbridge --data-dir=/data --config /etc/liftbridge.conf --nats-servers=nats://nats.liftbridge.svc:4222 --id="$HOSTNAME"
-fi
diff --git a/k8s/empty/kustomization.yaml b/k8s/empty/kustomization.yaml
new file mode 100644
index 0000000..e69de29
diff --git a/k8s/entrypoint.sh b/k8s/entrypoint.sh
new file mode 100755
index 0000000..437f5d8
--- /dev/null
+++ b/k8s/entrypoint.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+CLUSTER_NAME=${CLUSTER_NAME:=cluster}
+
+if [[ "$HOSTNAME" =~ -0$ ]] && [[ "$SKIP_SEED" != "true" ]];
+then
+  echo "Running in liftbridge-0 as a bootstrap seed..."
+  liftbridge --data-dir=/data --config /etc/liftbridge.conf --raft-bootstrap-seed --id="${CLUSTER_NAME}-$HOSTNAME"
+else
+  echo "Running in $HOSTNAME..."
+  liftbridge --data-dir=/data --config /etc/liftbridge.conf --id="${CLUSTER_NAME}-$HOSTNAME"
+fi
diff --git a/server/api.go b/server/api.go
index 21bec49..30e4b48 100644
--- a/server/api.go
+++ b/server/api.go
@@ -5,12 +5,12 @@ import (
 	"fmt"
 	"time"
 
-	client "github.com/liftbridge-io/liftbridge-api/go"
 	"github.com/nats-io/nats.go"
 	"github.com/nats-io/nuid"
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/status"
 
+	client "github.com/liftbridge-io/liftbridge-api/go"
 	"github.com/liftbridge-io/liftbridge/server/commitlog"
 	"github.com/liftbridge-io/liftbridge/server/proto"
 )
diff --git a/server/health/health.go b/server/health/health.go
new file mode 100644
index 0000000..22f9844
--- /dev/null
+++ b/server/health/health.go
@@ -0,0 +1,26 @@
+package health
+
+import (
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/health"
+	"google.golang.org/grpc/health/grpc_health_v1"
+)
+
+const serviceName = "proto.API" // taken from compiled protobuf file api.go.pb (line 793)
+
+var server = health.NewServer()
+
+// Register the health service with a gRPC server.
+func Register(srv *grpc.Server) {
+	grpc_health_v1.RegisterHealthServer(srv, server)
+}
+
+// SetServing marks the service as healthy.
+func SetServing() {
+	server.SetServingStatus(serviceName, grpc_health_v1.HealthCheckResponse_SERVING)
+}
+
+// SetNotServing marks the service as unhealthy.
+func SetNotServing() {
+	server.SetServingStatus(serviceName, grpc_health_v1.HealthCheckResponse_NOT_SERVING)
+}
diff --git a/server/metadata.go b/server/metadata.go
index 14986a8..61ec306 100644
--- a/server/metadata.go
+++ b/server/metadata.go
@@ -8,12 +8,12 @@ import (
 	"time"
 
 	"github.com/hashicorp/raft"
-	client "github.com/liftbridge-io/liftbridge-api/go"
 	"github.com/nats-io/nats.go"
 	"github.com/pkg/errors"
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/status"
 
+	client "github.com/liftbridge-io/liftbridge-api/go"
 	"github.com/liftbridge-io/liftbridge/server/proto"
 )
 
diff --git a/server/partition.go b/server/partition.go
index 2d33915..a5e8e98 100644
--- a/server/partition.go
+++ b/server/partition.go
@@ -11,10 +11,10 @@ import (
 	"time"
 
 	"github.com/Workiva/go-datastructures/queue"
-	client "github.com/liftbridge-io/liftbridge-api/go"
 	"github.com/nats-io/nats.go"
 	"github.com/pkg/errors"
 
+	client "github.com/liftbridge-io/liftbridge-api/go"
 	"github.com/liftbridge-io/liftbridge/server/commitlog"
 	"github.com/liftbridge-io/liftbridge/server/proto"
 )
diff --git a/server/proto/message.go b/server/proto/message.go
index 527b2e2..af81350 100644
--- a/server/proto/message.go
+++ b/server/proto/message.go
@@ -1,6 +1,8 @@
 package proto
 
-import client "github.com/liftbridge-io/liftbridge-api/go"
+import (
+	client "github.com/liftbridge-io/liftbridge-api/go"
+)
 
 // Message is the serialized object that gets written to the log.
 type Message struct {
diff --git a/server/server.go b/server/server.go
index 469000f..77d31ac 100644
--- a/server/server.go
+++ b/server/server.go
@@ -17,12 +17,13 @@ import (
 	"time"
 
 	"github.com/hashicorp/raft"
-	client "github.com/liftbridge-io/liftbridge-api/go"
 	"github.com/nats-io/nats.go"
 	"github.com/pkg/errors"
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/credentials"
 
+	client "github.com/liftbridge-io/liftbridge-api/go"
+	"github.com/liftbridge-io/liftbridge/server/health"
 	"github.com/liftbridge-io/liftbridge/server/logger"
 	"github.com/liftbridge-io/liftbridge/server/proto"
 )
@@ -167,11 +168,13 @@ func (s *Server) Start() (err error) {
 // Stop will attempt to gracefully shut the Server down by signaling the stop
 // and waiting for all goroutines to return.
 func (s *Server) Stop() error {
+	health.SetNotServing()
 	s.mu.Lock()
 	if s.shutdown {
 		s.mu.Unlock()
 		return nil
 	}
+
 	s.logger.Info("Shutting down...")
 
 	close(s.shutdownCh)
@@ -351,10 +354,14 @@ func (s *Server) startAPIServer() error {
 	api := grpc.NewServer(opts...)
 	s.api = api
 	client.RegisterAPIServer(api, &apiServer{s})
+
+	health.Register(api)
+
 	s.mu.Lock()
 	s.running = true
 	s.mu.Unlock()
 	s.startGoroutine(func() {
+		health.SetServing()
 		err := api.Serve(s.listener)
 		s.mu.Lock()
 		s.running = false
diff --git a/skaffold.yaml b/skaffold.yaml
index eea68e3..8b5110f 100644
--- a/skaffold.yaml
+++ b/skaffold.yaml
@@ -6,7 +6,7 @@ build:
     - image: liftbridge
       context: .
       docker:
-        dockerfile: k8s/dev/Dockerfile.kind
+        dockerfile: k8s/Dockerfile.k8s
     - image: natsboard
       context: dev/
       docker:
@@ -16,9 +16,7 @@ build:
   local:
     push: false
     useBuildkit: true
-deploy:
-  kustomize:
-    path: ./k8s/
+deploy: {}
 
 profiles:
   - name: dev
@@ -29,3 +27,17 @@ profiles:
       kubeContext: kubernetes-admin@kind
       kustomize:
         path: ./k8s/dev
+  - name: deploy-k8s-image
+    build:
+      local:
+        push: true
+      artifacts:
+        - image: liftbridge-io/liftbridge-k8s
+          context: .
+          docker:
+            dockerfile: k8s/Dockerfile.k8s
+      tagPolicy:
+        gitCommit: {}
+    deploy:
+      kustomize:
+        path: ./k8s/empty


Test patch:
diff --git a/server/api_test.go b/server/api_test.go
index 85e36db..99726d8 100644
--- a/server/api_test.go
+++ b/server/api_test.go
@@ -7,13 +7,14 @@ import (
 	"testing"
 	"time"
 
-	lift "github.com/liftbridge-io/go-liftbridge"
-	"github.com/liftbridge-io/liftbridge-api/go"
 	natsdTest "github.com/nats-io/nats-server/v2/test"
 	"github.com/stretchr/testify/require"
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/status"
+
+	lift "github.com/liftbridge-io/go-liftbridge"
+	proto "github.com/liftbridge-io/liftbridge-api/go"
 )
 
 type message struct {
diff --git a/server/partition_test.go b/server/partition_test.go
index 55cdb38..3b354e4 100644
--- a/server/partition_test.go
+++ b/server/partition_test.go
@@ -5,11 +5,11 @@ import (
 	"time"
 
 	"github.com/Workiva/go-datastructures/queue"
-	client "github.com/liftbridge-io/liftbridge-api/go"
 	natsdTest "github.com/nats-io/nats-server/v2/test"
 	"github.com/nats-io/nats.go"
 	"github.com/stretchr/testify/require"
 
+	client "github.com/liftbridge-io/liftbridge-api/go"
 	"github.com/liftbridge-io/liftbridge/server/proto"
 )
 
diff --git a/server/replicator_test.go b/server/replicator_test.go
index c1c32af..b501e5b 100644
--- a/server/replicator_test.go
+++ b/server/replicator_test.go
@@ -6,12 +6,12 @@ import (
 	"testing"
 	"time"
 
-	lift "github.com/liftbridge-io/go-liftbridge"
-	"github.com/liftbridge-io/liftbridge-api/go"
 	natsdTest "github.com/nats-io/nats-server/v2/test"
 	"github.com/nats-io/nats.go"
 	"github.com/stretchr/testify/require"
 
+	lift "github.com/liftbridge-io/go-liftbridge"
+	proto "github.com/liftbridge-io/liftbridge-api/go"
 	internal "github.com/liftbridge-io/liftbridge/server/proto"
 )
 
diff --git a/server/server_test.go b/server/server_test.go
index b29ae26..9028b6b 100644
--- a/server/server_test.go
+++ b/server/server_test.go
@@ -12,11 +12,13 @@ import (
 	"time"
 
 	lift "github.com/liftbridge-io/go-liftbridge"
-	"github.com/liftbridge-io/liftbridge-api/go"
 	natsdTest "github.com/nats-io/nats-server/v2/test"
 	log "github.com/sirupsen/logrus"
 	"github.com/stretchr/testify/require"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/health/grpc_health_v1"
 
+	proto "github.com/liftbridge-io/liftbridge-api/go"
 	internal "github.com/liftbridge-io/liftbridge/server/proto"
 )
 
@@ -260,6 +262,34 @@ func TestDurableServerID(t *testing.T) {
 	}
 }
 
+// Ensure server starts the gRPC health service correctly.
+func TestHealthServerStartedCorrectly(t *testing.T) {
+	defer cleanupStorage(t)
+
+	// Use a central NATS server.
+	ns := natsdTest.RunDefaultServer()
+	defer ns.Shutdown()
+
+	// Configure the first server as a seed.
+	s1Config := getTestConfig("a", true, 20000)
+	s1 := runServerWithConfig(t, s1Config)
+	defer s1.Stop()
+
+	// Configure second server which should automatically join the first.
+	s2Config := getTestConfig("b", false, 20001)
+	s2 := runServerWithConfig(t, s2Config)
+	defer s2.Stop()
+
+	conn, err := grpc.Dial("127.0.0.1:20000", []grpc.DialOption{grpc.WithInsecure()}...)
+	require.NoError(t, err)
+	healthClient := grpc_health_v1.NewHealthClient(conn)
+	healthCheckReply, err := healthClient.Check(context.Background(), &grpc_health_v1.HealthCheckRequest{Service: "proto.API"})
+	require.NoError(t, err)
+	if healthCheckReply.Status != grpc_health_v1.HealthCheckResponse_SERVING {
+		t.Fatalf("API service is not healthy when it should be: %v", err)
+	}
+}
+
 // Ensure starting a cluster with auto configuration works when we start one
 // node in bootstrap mode.
 func TestBootstrapAutoConfig(t *testing.T) {