Repository: hazelcast/hazelcast-kubernetes (java)

Issue:
Label PODs instead of services
Filtering Hazelcast members by labeling services is weird. The more straightforward approach would be to just label PODs and make the labeling services deprecated.

Related to : #113 

Pull request description:
discover hazelcast members via pod labels
With this PR, our discovery plugin can filter hazelcast members via pod labels. This enhancement was tested at GKE cluster. If you want to test it, you can use `hasancelik/hazelcast:3.12.1-pod` docker image which includes related snapshot jars.

I am not sure should we make `service label` based discovery mech. deprecated, as @henrikarro mentioned [at related issue](https://github.com/hazelcast/hazelcast-kubernetes/issues/115#issuecomment-515350389), these parameter are still useful for some use-cases. To prevent `service-label  pod-label` confusion at user side, we can prepare a detailed README with some `yaml` samples. WDYT @leszko?

fix #115 

TODO:

- [x] add unit test

- [x] update README

Patch:
diff --git a/README.md b/README.md
index a2a4834..529bbab 100644
--- a/README.md
+++ b/README.md
@@ -123,6 +123,7 @@ There are several properties to configure the plugin, all of them are optional.
  * `namespace`: Kubernetes Namespace where Hazelcast is running; if not specified, the value is taken from the environment variables `KUBERNETES_NAMESPACE` or `OPENSHIFT_BUILD_NAMESPACE`
  * `service-name`: service name used to scan only PODs connected to the given service; if not specified, then all PODs in the namespace are checked
  * `service-label-name`, `service-label-value`: service label and value used to tag services that should form the Hazelcast cluster together
+ * `pod-label-name`, `pod-label-value`: pod label and value used to tag pods that should form the Hazelcast cluster together.
  * `resolve-not-ready-addresses`: if set to `true`, it checks also the addresses of PODs which are not ready; `false` by default
  * `use-node-name-as-external-address`: if set to `true`, uses the node name to connect to a `NodePort` service instead of looking up the external IP using the API; `false` by default
  * `kubernetes-api-retries`: number of retries in case of issues while connecting to Kubernetes API; defaults to `3` 
@@ -131,7 +132,7 @@ There are several properties to configure the plugin, all of them are optional.
  * `ca-certificate`: CA Certificate for Kubernetes API; if not specified, the value is taken from the file `/var/run/secrets/kubernetes.io/serviceaccount/ca.crt`
  * `service-port`: endpoint port of the service; if specified with a value greater than `0`, it overrides the default; `0` by default
  
-You should use either `service-name` or (`service-label-name` and `service-label-value`), specifying all 3 parameters does not make sense.
+You can use one of `service-name`,`service-label`(*service-label-name,service-label-value*) and `pod-label`(*pod-label-name, pod-label-value*) based discovery mechanisms, configuring two of them at once does not make sense.
 
 *Note*: If you don't specify any property at all, then the Hazelcast cluster is formed using all PODs in your current namespace. In other words, you can look at the properties as a grouping feature if you want to have multiple Hazelcast clusters in one namespace.
 
diff --git a/pom.xml b/pom.xml
index c6420f0..f353f8e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -50,7 +50,7 @@
     <mockito.version>1.10.19</mockito.version>
     <powermock.version>1.7.3</powermock.version>
 
-    <hazelcast.version>3.11</hazelcast.version>
+    <hazelcast.version>3.12.1</hazelcast.version>
 
     <maven.checkstyle.plugin.version>2.15</maven.checkstyle.plugin.version>
     <maven.findbugs.plugin.version>3.0.1</maven.findbugs.plugin.version>
diff --git a/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategy.java b/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategy.java
index c5a7f14..28ea8b1 100644
--- a/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategy.java
+++ b/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategy.java
@@ -48,7 +48,9 @@ final class HazelcastKubernetesDiscoveryStrategy
                     config.getServiceDnsTimeout());
         } else {
             endpointResolver = new KubernetesApiEndpointResolver(logger, config.getServiceName(), config.getServicePort(),
-                    config.getServiceLabelName(), config.getServiceLabelValue(), config.isResolveNotReadyAddresses(), client);
+                    config.getServiceLabelName(), config.getServiceLabelValue(),
+                    config.getPodLabelName(), config.getPodLabelValue(),
+                    config.isResolveNotReadyAddresses(), client);
         }
 
         logger.info("Kubernetes Discovery activated with mode: " + config.getMode().name());
diff --git a/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactory.java b/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactory.java
index 5f3a7d0..1019f8e 100644
--- a/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactory.java
+++ b/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactory.java
@@ -43,6 +43,8 @@ public class HazelcastKubernetesDiscoveryStrategyFactory
                 KubernetesProperties.SERVICE_LABEL_NAME,
                 KubernetesProperties.SERVICE_LABEL_VALUE,
                 KubernetesProperties.NAMESPACE,
+                KubernetesProperties.POD_LABEL_NAME,
+                KubernetesProperties.POD_LABEL_VALUE,
                 KubernetesProperties.RESOLVE_NOT_READY_ADDRESSES,
                 KubernetesProperties.USE_NODE_NAME_AS_EXTERNAL_ADDRESS,
                 KubernetesProperties.KUBERNETES_API_RETIRES,
diff --git a/src/main/java/com/hazelcast/kubernetes/KubernetesApiEndpointResolver.java b/src/main/java/com/hazelcast/kubernetes/KubernetesApiEndpointResolver.java
index 7085aaf..4ee81c9 100644
--- a/src/main/java/com/hazelcast/kubernetes/KubernetesApiEndpointResolver.java
+++ b/src/main/java/com/hazelcast/kubernetes/KubernetesApiEndpointResolver.java
@@ -33,11 +33,14 @@ class KubernetesApiEndpointResolver
     private final String serviceName;
     private final String serviceLabel;
     private final String serviceLabelValue;
+    private final String podLabel;
+    private final String podLabelValue;
     private final Boolean resolveNotReadyAddresses;
     private final int port;
     private final KubernetesClient client;
 
-    KubernetesApiEndpointResolver(ILogger logger, String serviceName, int port, String serviceLabel, String serviceLabelValue,
+    KubernetesApiEndpointResolver(ILogger logger, String serviceName, int port,
+                                  String serviceLabel, String serviceLabelValue, String podLabel, String podLabelValue,
                                   Boolean resolveNotReadyAddresses, KubernetesClient client) {
 
         super(logger);
@@ -46,6 +49,8 @@ class KubernetesApiEndpointResolver
         this.port = port;
         this.serviceLabel = serviceLabel;
         this.serviceLabelValue = serviceLabelValue;
+        this.podLabel = podLabel;
+        this.podLabelValue = podLabelValue;
         this.resolveNotReadyAddresses = resolveNotReadyAddresses;
         this.client = client;
     }
@@ -53,9 +58,14 @@ class KubernetesApiEndpointResolver
     @Override
     List<DiscoveryNode> resolve() {
         if (serviceName != null && !serviceName.isEmpty()) {
+            logger.fine("Using service name to discover nodes.");
             return getSimpleDiscoveryNodes(client.endpointsByName(serviceName));
         } else if (serviceLabel != null && !serviceLabel.isEmpty()) {
-            return getSimpleDiscoveryNodes(client.endpointsByLabel(serviceLabel, serviceLabelValue));
+            logger.fine("Using service label to discover nodes.");
+            return getSimpleDiscoveryNodes(client.endpointsByServiceLabel(serviceLabel, serviceLabelValue));
+        } else if (podLabel != null && !podLabel.isEmpty()) {
+            logger.fine("Using pod label to discover nodes.");
+            return getSimpleDiscoveryNodes(client.endpointsByPodLabel(podLabel, podLabelValue));
         }
         return getSimpleDiscoveryNodes(client.endpoints());
     }
diff --git a/src/main/java/com/hazelcast/kubernetes/KubernetesClient.java b/src/main/java/com/hazelcast/kubernetes/KubernetesClient.java
index bb0af5a..438210e 100644
--- a/src/main/java/com/hazelcast/kubernetes/KubernetesClient.java
+++ b/src/main/java/com/hazelcast/kubernetes/KubernetesClient.java
@@ -92,7 +92,7 @@ class KubernetesClient {
      * @return all POD addresses from the specified {@code namespace} filtered by the label
      * @see <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#list-143">Kubernetes Endpoint API</a>
      */
-    List<Endpoint> endpointsByLabel(String serviceLabel, String serviceLabelValue) {
+    List<Endpoint> endpointsByServiceLabel(String serviceLabel, String serviceLabelValue) {
         try {
             String param = String.format("labelSelector=%s=%s", serviceLabel, serviceLabelValue);
             String urlString = String.format("%s/api/v1/namespaces/%s/endpoints?%s", kubernetesMaster, namespace, param);
@@ -118,6 +118,26 @@ class KubernetesClient {
         }
     }
 
+    /**
+     * Retrieves POD addresses for all services in the specified {@code namespace} filtered by {@code podLabel}
+     * and {@code podLabelValue}.
+     *
+     * @param podLabel      label used to filter responses
+     * @param podLabelValue label value used to filter responses
+     * @return all POD addresses from the specified {@code namespace} filtered by the label
+     * @see <a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.11/#list-143">Kubernetes Endpoint API</a>
+     */
+    List<Endpoint> endpointsByPodLabel(String podLabel, String podLabelValue) {
+        try {
+            String param = String.format("labelSelector=%s=%s", podLabel, podLabelValue);
+            String urlString = String.format("%s/api/v1/namespaces/%s/pods?%s", kubernetesMaster, namespace, param);
+            return enrichWithPublicAddresses(parsePodsList(callGet(urlString)));
+        } catch (RestClientException e) {
+            return handleKnownException(e);
+        }
+    }
+
+
     /**
      * Retrieves zone name for the specified {@code namespace} and the given {@code podName}.
      * <p>
diff --git a/src/main/java/com/hazelcast/kubernetes/KubernetesConfig.java b/src/main/java/com/hazelcast/kubernetes/KubernetesConfig.java
index 313827e..14349f8 100644
--- a/src/main/java/com/hazelcast/kubernetes/KubernetesConfig.java
+++ b/src/main/java/com/hazelcast/kubernetes/KubernetesConfig.java
@@ -34,6 +34,8 @@ import static com.hazelcast.kubernetes.KubernetesProperties.KUBERNETES_CA_CERTIF
 import static com.hazelcast.kubernetes.KubernetesProperties.KUBERNETES_MASTER_URL;
 import static com.hazelcast.kubernetes.KubernetesProperties.KUBERNETES_SYSTEM_PREFIX;
 import static com.hazelcast.kubernetes.KubernetesProperties.NAMESPACE;
+import static com.hazelcast.kubernetes.KubernetesProperties.POD_LABEL_NAME;
+import static com.hazelcast.kubernetes.KubernetesProperties.POD_LABEL_VALUE;
 import static com.hazelcast.kubernetes.KubernetesProperties.RESOLVE_NOT_READY_ADDRESSES;
 import static com.hazelcast.kubernetes.KubernetesProperties.SERVICE_DNS;
 import static com.hazelcast.kubernetes.KubernetesProperties.SERVICE_DNS_TIMEOUT;
@@ -46,6 +48,7 @@ import static com.hazelcast.kubernetes.KubernetesProperties.USE_NODE_NAME_AS_EXT
 /**
  * Responsible for fetching, parsing, and validating Hazelcast Kubernetes Discovery Strategy input properties.
  */
+@SuppressWarnings({"checkstyle:npathcomplexity", "checkstyle:cyclomaticcomplexity"})
 final class KubernetesConfig {
     private static final String DEFAULT_MASTER_URL = "https://kubernetes.default.svc";
     private static final int DEFAULT_SERVICE_DNS_TIMEOUT_SECONDS = 5;
@@ -60,6 +63,8 @@ final class KubernetesConfig {
     private final String serviceLabelName;
     private final String serviceLabelValue;
     private final String namespace;
+    private final String podLabelName;
+    private final String podLabelValue;
     private final boolean resolveNotReadyAddresses;
     private final boolean useNodeNameAsExternalAddress;
     private final int kubernetesApiRetries;
@@ -78,6 +83,8 @@ final class KubernetesConfig {
         this.serviceLabelName = getOrNull(properties, KUBERNETES_SYSTEM_PREFIX, SERVICE_LABEL_NAME);
         this.serviceLabelValue = getOrDefault(properties, KUBERNETES_SYSTEM_PREFIX, SERVICE_LABEL_VALUE, "true");
         this.namespace = getOrDefault(properties, KUBERNETES_SYSTEM_PREFIX, NAMESPACE, getNamespaceOrDefault());
+        this.podLabelName = getOrNull(properties, KUBERNETES_SYSTEM_PREFIX, POD_LABEL_NAME);
+        this.podLabelValue = getOrNull(properties, KUBERNETES_SYSTEM_PREFIX, POD_LABEL_VALUE);
         this.resolveNotReadyAddresses = getOrDefault(properties, KUBERNETES_SYSTEM_PREFIX, RESOLVE_NOT_READY_ADDRESSES, false);
         this.useNodeNameAsExternalAddress
                 = getOrDefault(properties, KUBERNETES_SYSTEM_PREFIX, USE_NODE_NAME_AS_EXTERNAL_ADDRESS, false);
@@ -198,16 +205,25 @@ final class KubernetesConfig {
     }
 
     private void validateConfig() {
-        if (serviceDns != null && (serviceName != null || serviceLabelName != null)) {
+        if (serviceDns != null && (serviceName != null || serviceLabelName != null || podLabelName != null)) {
             throw new InvalidConfigurationException(
-                    String.format("Properties '%s' and ('%s' or '%s') cannot be defined at the same time",
-                            SERVICE_DNS.key(), SERVICE_NAME.key(), SERVICE_LABEL_NAME.key()));
+                    String.format("Properties '%s' and ('%s' or '%s' or %s) cannot be defined at the same time",
+                            SERVICE_DNS.key(), SERVICE_NAME.key(), SERVICE_LABEL_NAME.key(), POD_LABEL_NAME.key()));
         }
         if (serviceName != null && serviceLabelName != null) {
             throw new InvalidConfigurationException(
                     String.format("Properties '%s' and '%s' cannot be defined at the same time",
                             SERVICE_NAME.key(), SERVICE_LABEL_NAME.key()));
-
+        }
+        if (serviceName != null && podLabelName != null) {
+            throw new InvalidConfigurationException(
+                    String.format("Properties '%s' and '%s' cannot be defined at the same time",
+                            SERVICE_NAME.key(), POD_LABEL_NAME.key()));
+        }
+        if (serviceLabelName != null && podLabelName != null) {
+            throw new InvalidConfigurationException(
+                    String.format("Properties '%s' and '%s' cannot be defined at the same time",
+                            SERVICE_LABEL_NAME.key(), POD_LABEL_NAME.key()));
         }
         if (serviceDnsTimeout < 0) {
             throw new InvalidConfigurationException(
@@ -255,6 +271,14 @@ final class KubernetesConfig {
         return namespace;
     }
 
+    public String getPodLabelName() {
+        return podLabelName;
+    }
+
+    public String getPodLabelValue() {
+        return podLabelValue;
+    }
+
     boolean isResolveNotReadyAddresses() {
         return resolveNotReadyAddresses;
     }
@@ -293,6 +317,8 @@ final class KubernetesConfig {
                 + "service-label: " + serviceLabelName + ", "
                 + "service-label-value: " + serviceLabelValue + ", "
                 + "namespace: " + namespace + ", "
+                + "pod-label: " + podLabelName + ", "
+                + "pod-label-value: " + podLabelValue + ", "
                 + "resolve-not-ready-addresses: " + resolveNotReadyAddresses + ", "
                 + "use-node-name-as-external-address: " + useNodeNameAsExternalAddress + ", "
                 + "kubernetes-api-retries: " + kubernetesApiRetries + ", "
diff --git a/src/main/java/com/hazelcast/kubernetes/KubernetesProperties.java b/src/main/java/com/hazelcast/kubernetes/KubernetesProperties.java
index 1db1347..21df0b8 100644
--- a/src/main/java/com/hazelcast/kubernetes/KubernetesProperties.java
+++ b/src/main/java/com/hazelcast/kubernetes/KubernetesProperties.java
@@ -85,6 +85,17 @@ public final class KubernetesProperties {
      */
     public static final PropertyDefinition NAMESPACE = property("namespace", STRING);
 
+    /**
+     * <p>Configuration key: <tt>pod-label-name</tt></p>
+     * Defines the pod label to lookup through the Service Discovery REST API of Kubernetes.
+     */
+    public static final PropertyDefinition POD_LABEL_NAME = property("pod-label-name", STRING);
+    /**
+     * <p>Configuration key: <tt>pod-label-value</tt></p>
+     * Defines the pod label value to lookup through the Service Discovery REST API of Kubernetes.
+     */
+    public static final PropertyDefinition POD_LABEL_VALUE = property("pod-label-value", STRING);
+
     /**
      * <p>Configuration key: <tt>resolve-not-ready-addresses</tt></p>
      * Defines if not ready addresses should be evaluated to be discovered on startup.


Test patch:
diff --git a/src/test/java/com/hazelcast/kubernetes/KubernetesApiEndpointResolverTest.java b/src/test/java/com/hazelcast/kubernetes/KubernetesApiEndpointResolverTest.java
index 683cb4d..2f8d2cc 100644
--- a/src/test/java/com/hazelcast/kubernetes/KubernetesApiEndpointResolverTest.java
+++ b/src/test/java/com/hazelcast/kubernetes/KubernetesApiEndpointResolverTest.java
@@ -40,8 +40,10 @@ import static org.mockito.BDDMockito.given;
 public class KubernetesApiEndpointResolverTest {
     private static final ILogger LOGGER = new NoLogFactory().getLogger("no");
     private static final String SERVICE_NAME = "serviceName";
-    private static final String SERVICE_LABEL = "theLabel";
+    private static final String SERVICE_LABEL = "serviceLabel";
     private static final String SERVICE_LABEL_VALUE = "serviceLabelValue";
+    private static final String POD_LABEL = "podLabel";
+    private static final String POD_LABEL_VALUE = "podLabelValue";
     private static final Boolean RESOLVE_NOT_READY_ADDRESSES = true;
 
     @Mock
@@ -59,7 +61,7 @@ public class KubernetesApiEndpointResolverTest {
         List<Endpoint> endpoints = Collections.<Endpoint>emptyList();
         given(client.endpoints()).willReturn(endpoints);
 
-        KubernetesApiEndpointResolver sut = new KubernetesApiEndpointResolver(LOGGER, null, 0, null, null, null, client);
+        KubernetesApiEndpointResolver sut = new KubernetesApiEndpointResolver(LOGGER, null, 0, null, null, null, null, null, client);
 
         // when
         List<DiscoveryNode> nodes = sut.resolve();
@@ -83,7 +85,7 @@ public class KubernetesApiEndpointResolverTest {
         List<Endpoint> endpoints = createEndpoints(1);
         given(client.endpointsByName(SERVICE_NAME)).willReturn(endpoints);
 
-        KubernetesApiEndpointResolver sut = new KubernetesApiEndpointResolver(LOGGER, SERVICE_NAME, port, null, null, null,
+        KubernetesApiEndpointResolver sut = new KubernetesApiEndpointResolver(LOGGER, SERVICE_NAME, port, null, null, null, null, null,
                 client);
 
         // when
@@ -98,10 +100,27 @@ public class KubernetesApiEndpointResolverTest {
     public void resolveWithServiceLabelWhenNodeWithServiceLabel() {
         // given
         List<Endpoint> endpoints = createEndpoints(2);
-        given(client.endpointsByLabel(SERVICE_LABEL, SERVICE_LABEL_VALUE)).willReturn(endpoints);
+        given(client.endpointsByServiceLabel(SERVICE_LABEL, SERVICE_LABEL_VALUE)).willReturn(endpoints);
 
         KubernetesApiEndpointResolver sut = new KubernetesApiEndpointResolver(LOGGER, null, 0, SERVICE_LABEL, SERVICE_LABEL_VALUE,
-                null, client);
+                null, null, null, client);
+
+        // when
+        List<DiscoveryNode> nodes = sut.resolve();
+
+        // then
+        assertEquals(1, nodes.size());
+        assertEquals(2, nodes.get(0).getPrivateAddress().getPort());
+    }
+
+    @Test
+    public void resolveWithPodLabelWhenNodeWithPodLabel() {
+        // given
+        List<Endpoint> endpoints = createEndpoints(2);
+        given(client.endpointsByPodLabel(POD_LABEL, POD_LABEL_VALUE)).willReturn(endpoints);
+
+        KubernetesApiEndpointResolver sut = new KubernetesApiEndpointResolver(LOGGER, null, 0, null, null,
+                POD_LABEL, POD_LABEL_VALUE, null, client);
 
         // when
         List<DiscoveryNode> nodes = sut.resolve();
@@ -118,7 +137,7 @@ public class KubernetesApiEndpointResolverTest {
         given(client.endpointsByName(SERVICE_NAME)).willReturn(endpoints);
 
         KubernetesApiEndpointResolver sut = new KubernetesApiEndpointResolver(LOGGER, SERVICE_NAME, 0, null, null,
-                RESOLVE_NOT_READY_ADDRESSES, client);
+                null, null, RESOLVE_NOT_READY_ADDRESSES, client);
 
         // when
         List<DiscoveryNode> nodes = sut.resolve();
@@ -133,7 +152,7 @@ public class KubernetesApiEndpointResolverTest {
         List<Endpoint> endpoints = createNotReadyEndpoints(2);
         given(client.endpointsByName(SERVICE_NAME)).willReturn(endpoints);
 
-        KubernetesApiEndpointResolver sut = new KubernetesApiEndpointResolver(LOGGER, SERVICE_NAME, 0, null, null, null,
+        KubernetesApiEndpointResolver sut = new KubernetesApiEndpointResolver(LOGGER, SERVICE_NAME, 0, null, null, null, null, null,
                 client);
 
         // when
diff --git a/src/test/java/com/hazelcast/kubernetes/KubernetesClientTest.java b/src/test/java/com/hazelcast/kubernetes/KubernetesClientTest.java
index 778b333..b0f7218 100644
--- a/src/test/java/com/hazelcast/kubernetes/KubernetesClientTest.java
+++ b/src/test/java/com/hazelcast/kubernetes/KubernetesClientTest.java
@@ -142,7 +142,7 @@ public class KubernetesClientTest {
     }
 
     @Test
-    public void endpointsByNamespaceAndLabel() {
+    public void endpointsByNamespaceAndServiceLabel() {
         // given
         //language=JSON
         String endpointsListResponse = "{\n"
@@ -190,7 +190,7 @@ public class KubernetesClientTest {
         stub(String.format("/api/v1/namespaces/%s/endpoints", NAMESPACE), queryParams, endpointsListResponse);
 
         // when
-        List<Endpoint> result = kubernetesClient.endpointsByLabel(serviceLabel, serviceLabelValue);
+        List<Endpoint> result = kubernetesClient.endpointsByServiceLabel(serviceLabel, serviceLabelValue);
 
         // then
         assertThat(format(result),
@@ -228,6 +228,71 @@ public class KubernetesClientTest {
         assertThat(format(result), containsInAnyOrder(ready("192.168.0.25", 5701), ready("172.17.0.5", 5702)));
     }
 
+    @Test
+    public void endpointsByNamespaceAndPodLabel() {
+        // given
+        //language=JSON
+        String podsListResponse = "{\n"
+                + "  \"kind\": \"PodList\",\n"
+                + "  \"items\": [\n"
+                + "    {\n"
+                + "      \"spec\": {\n"
+                + "        \"containers\": [\n"
+                + "          {\n"
+                + "            \"ports\": [\n"
+                + "              {\n"
+                + "                \"containerPort\": 5701\n"
+                + "              }\n"
+                + "            ]\n"
+                + "          }\n"
+                + "        ]\n"
+                + "      },\n"
+                + "      \"status\": {\n"
+                + "        \"podIP\": \"192.168.0.25\",\n"
+                + "        \"containerStatuses\": [\n"
+                + "          {\n"
+                + "            \"ready\": true\n"
+                + "          }\n"
+                + "        ]\n"
+                + "      }\n"
+                + "    },\n"
+                + "    {\n"
+                + "      \"spec\": {\n"
+                + "        \"containers\": [\n"
+                + "          {\n"
+                + "            \"ports\": [\n"
+                + "              {\n"
+                + "                \"containerPort\": 5702\n"
+                + "              }\n"
+                + "            ]\n"
+                + "          }\n"
+                + "        ]\n"
+                + "      },\n"
+                + "      \"status\": {\n"
+                + "        \"podIP\": \"172.17.0.5\",\n"
+                + "        \"containerStatuses\": [\n"
+                + "          {\n"
+                + "            \"ready\": true\n"
+                + "          }\n"
+                + "        ]\n"
+                + "      }\n"
+                + "    }\n"
+                + "  ]\n"
+                + "}";
+
+        String podLabel = "sample-pod-label";
+        String podLabelValue = "sample-pod-label-value";
+        Map<String, String> queryParams = singletonMap("labelSelector", String.format("%s=%s", podLabel, podLabelValue));
+        stub(String.format("/api/v1/namespaces/%s/pods", NAMESPACE, podLabel), queryParams, podsListResponse);
+
+        // when
+        List<Endpoint> result = kubernetesClient.endpointsByPodLabel(podLabel, podLabelValue);
+
+        // then
+        assertThat(format(result),
+                containsInAnyOrder(ready("192.168.0.25", 5701), ready("172.17.0.5", 5702)));
+    }
+
     @Test
     public void zoneBeta() {
         // given