Repository: hazelcast/hazelcast-kubernetes
Pull request #125 (2019-01-08 16:54:48)

Add "ca-certificate" property to define the custom Kubernetes Master CA
This is the missing part of the external authentication to Kubernetes API (Kubernetes Master). Now you can use Kubernetes API from outside the cluster by defining the following properties:
- `kubernetes-master` - address of Kubernetes Master
- `api-token` - API Access Token to Kubernetes Master
- `ca-certificate` - CA Certificate to Kubernetes Master

The implementation allows you to specify the certificate by pasting the string directly into your `hazelcast.xml`, which looks quite ugly, like:

```
<ca-certificate>
      -----BEGIN CERTIFICATE-----
      MIIDDDCCAfSgAwIBAgIRAOlepQ2R2h/eyHYayR0AfLIwDQYJKoZIhvcNAQELBQAw
      LzEtMCsGA1UEAxMkMzEyMDU1ZmYtZDhmZi00OGJiLTlhYmItMmZiZDFkZjlmY2Iz
      MB4XDTE5MDEwODE0NDIxMVoXDTI0MDEwNzE1NDIxMVowLzEtMCsGA1UEAxMkMzEy
      MDU1ZmYtZDhmZi00OGJiLTlhYmItMmZiZDFkZjlmY2IzMIIBIjANBgkqhkiG9w0B
      AQEFAAOCAQ8AMIIBCgKCAQEAwPLdrqmvKJwfHMF8mz79Jmh9uevFzQVL1qP/KBUp
      CUKe6C1cacKByy5RTQWuCHrMi6scxWmjwnpaFyHbArD3P7KD6OIBqDHuvxezk0NK
      YVJRGTKje/ShdGdLRYQvF8VZtdA6WmXusvc2OChnrcf+wYJaGeP/k2koOEd/JbKT
      TSMUm3H8BtQUeZ9RZIUuAyqHIXZxjM2K4yZOnT1cmRGNjYcp4ABkNYWE0xTMUT19
      LjEG/+Ka6SmyyBbsVVFEIuJnqAKhXryP5QFwRYrZnxNTYu1PMxGoNeIo89ZrTRKq
      s3ZSyZNbnky65+KViiWWqa8gkpd3EXHH7ZD+Z6Ka2iwt9QIDAQABoyMwITAOBgNV
      HQ8BAf8EBAMCAgQwDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEA
      Xol4SW/dNzzk9pDhiGNBiUl+Z/KBNgDpuYs87APLTE421tSvN/npkYUqDmHfr8OA
      G3xqFXviYpKe3j1cixFFwV28MRkGyhYicgnojv6dInzjVV5QxjkB0yRl8OSh48+h
      V2hD4KZx79LzlOZTP3IEknpcQpkrYrBDd+wPhmq9pP+gTJm7W9hVr02WSSbpHl67
      NeRbjUoML2n4ZoidQ39rl6MP1kEW0BwlTNXGHHqqF3TzbT+/qywrPJntaL2oh7lw
      NfqYkzJp69EylHztVfyNuwH6p6yg7PH9yBsRR2rZmXYtz0XQoGScHG5nNzrDbpMr
      8qhxxHgZb4K85CbIQ31AZg==
      -----END CERTIFICATE-----
</ca-certificate>
```

The other option would be to have a property with the path to the file containing the Certificate. But, then, it's more complex to configure, plus it's not consistent with what we currently have in case of the `api-token` property. @googlielmo @mesutcelik , please tell me your thoughts.

devOpsHazelcast: Test PASSed.

devOpsHazelcast: Test PASSed.

mesutcelik: I have tried that by connecting from laptop to a GKE hosted k8s cluster. I receive following error.

```
Message: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:default:default\" cannot list pods in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403},
	at com.hazelcast.kubernetes.DefaultKubernetesClient.callGet(DefaultKubernetesClient.java:116)
```
mesutcelik: It would be good to mention this PR in README as well.
hasancelik: verify
leszko: > I have tried that by connecting from laptop to a GKE hosted k8s cluster. I receive following error.
> 
> ```
> Message: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:default:default\" cannot list pods in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403},
> 	at com.hazelcast.kubernetes.DefaultKubernetesClient.callGet(DefaultKubernetesClient.java:116)
> ```

What access-token and ca-certificate did you use?
leszko: > README

Yes, good idea. I'll create a separate PR for that (we should merge it after this feature is released).
Plus, I plan to write a blog post, because it may not be so straightforward how to create/fetch access-token and ca-certificate for the k8 cluster.
mesutcelik: I think it is good to add how to fetch token/certifacte into README.

I used this to get the token
`TOKEN=$(kubectl describe secret $(kubectl get secrets | grep "^default" | cut -f1 -d ' ') | grep -E '^token' | cut -f2 -d':' | tr -d " ")`

I don't remember how  I downloaded certificate. 

Do we need both certificate and API token configured to make the calls?
[Here](https://kubernetes.io/docs/tasks/administer-cluster/access-cluster-api/#without-kubectl-proxy) it says APItoken is sufficient but I still get the same error when I call following.
`curl $APISERVER/v1/namespaces/default/pods --header "Authorization: Bearer $TOKEN" --insecure`

I think I need a better README to make all working for me 😄 


mesutcelik: I will review again a new README PR submitted. Thanks @leszko 

Patch:
diff --git a/src/main/java/com/hazelcast/kubernetes/DefaultKubernetesClient.java b/src/main/java/com/hazelcast/kubernetes/DefaultKubernetesClient.java
index 52783a1..8e660c7 100644
--- a/src/main/java/com/hazelcast/kubernetes/DefaultKubernetesClient.java
+++ b/src/main/java/com/hazelcast/kubernetes/DefaultKubernetesClient.java
@@ -21,15 +21,12 @@ import com.hazelcast.internal.json.JsonArray;
 import com.hazelcast.internal.json.JsonObject;
 import com.hazelcast.internal.json.JsonValue;
 import com.hazelcast.nio.IOUtil;
-import edu.umd.cs.findbugs.annotations.SuppressFBWarnings;
 
 import javax.net.ssl.HttpsURLConnection;
 import javax.net.ssl.SSLContext;
 import javax.net.ssl.SSLSocketFactory;
 import javax.net.ssl.TrustManagerFactory;
-import java.io.BufferedInputStream;
-import java.io.File;
-import java.io.FileInputStream;
+import java.io.ByteArrayInputStream;
 import java.io.IOException;
 import java.io.InputStream;
 import java.net.HttpURLConnection;
@@ -64,10 +61,12 @@ class DefaultKubernetesClient
 
     private final String kubernetesMaster;
     private final String apiToken;
+    private final String caCertificate;
 
-    DefaultKubernetesClient(String kubernetesMaster, String apiToken) {
+    DefaultKubernetesClient(String kubernetesMaster, String apiToken, String caCertificate) {
         this.kubernetesMaster = kubernetesMaster;
         this.apiToken = apiToken;
+        this.caCertificate = caCertificate;
     }
 
     @Override
@@ -275,7 +274,7 @@ class DefaultKubernetesClient
     /**
      * Builds SSL Socket Factory with the public CA Certificate from Kubernetes Master.
      */
-    private static SSLSocketFactory buildSslSocketFactory() {
+    private SSLSocketFactory buildSslSocketFactory() {
         try {
             KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
             keyStore.load(null, null);
@@ -294,23 +293,18 @@ class DefaultKubernetesClient
     }
 
     /**
-     * Generates CA Certificate from the default CA Cert file (which always exists in the Kubernetes POD).
+     * Generates CA Certificate from the default CA Cert file or from the externally provided "ca-certificate" property.
      */
-    private static Certificate generateCertificate()
+    private Certificate generateCertificate()
             throws IOException, CertificateException {
         InputStream caInput = null;
         try {
             CertificateFactory cf = CertificateFactory.getInstance("X.509");
-            caInput = new BufferedInputStream(new FileInputStream(new File(caCertPath())));
+            caInput = new ByteArrayInputStream(caCertificate.getBytes("UTF-8"));
             return cf.generateCertificate(caInput);
         } finally {
             IOUtil.closeResource(caInput);
         }
     }
 
-    @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
-    private static String caCertPath() {
-        return "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt";
-    }
-
 }
diff --git a/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategy.java b/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategy.java
index 8b80407..d6e2bb2 100644
--- a/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategy.java
+++ b/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategy.java
@@ -36,6 +36,7 @@ import java.util.List;
 import java.util.Map;
 
 import static com.hazelcast.kubernetes.KubernetesProperties.KUBERNETES_API_TOKEN;
+import static com.hazelcast.kubernetes.KubernetesProperties.KUBERNETES_CA_CERTIFICATE;
 import static com.hazelcast.kubernetes.KubernetesProperties.KUBERNETES_MASTER_URL;
 import static com.hazelcast.kubernetes.KubernetesProperties.KUBERNETES_SYSTEM_PREFIX;
 import static com.hazelcast.kubernetes.KubernetesProperties.NAMESPACE;
@@ -77,6 +78,10 @@ final class HazelcastKubernetesDiscoveryStrategy
         if (apiToken == null) {
             apiToken = readAccountToken();
         }
+        String caCertificate = getOrDefault(properties, KUBERNETES_SYSTEM_PREFIX, KUBERNETES_CA_CERTIFICATE, null);
+        if (caCertificate == null) {
+            caCertificate = readCaCertificate();
+        }
 
         logger.info("Kubernetes Discovery properties: { "
                 + "service-dns: " + serviceDns + ", "
@@ -89,8 +94,7 @@ final class HazelcastKubernetesDiscoveryStrategy
                 + "resolve-not-ready-addresses: " + resolveNotReadyAddresses + ", "
                 + "kubernetes-master: " + kubernetesMaster + "}");
 
-        client = buildKubernetesClient(apiToken, kubernetesMaster);
-        EndpointResolver endpointResolver;
+        client = buildKubernetesClient(kubernetesMaster, apiToken, caCertificate);
         if (serviceDns != null) {
             endpointResolver = new DnsEndpointResolver(logger, serviceDns, port, serviceDnsTimeout);
         } else {
@@ -98,7 +102,6 @@ final class HazelcastKubernetesDiscoveryStrategy
                     namespace, resolveNotReadyAddresses, client);
         }
         logger.info("Kubernetes Discovery activated resolver: " + endpointResolver.getClass().getSimpleName());
-        this.endpointResolver = endpointResolver;
     }
 
     private String getNamespaceOrDefault() {
@@ -236,11 +239,8 @@ final class HazelcastKubernetesDiscoveryStrategy
         }
     }
 
-    private KubernetesClient buildKubernetesClient(String token, String kubernetesMaster) {
-        if (StringUtil.isNullOrEmpty(token)) {
-            token = readAccountToken();
-        }
-        return new RetryKubernetesClient(new DefaultKubernetesClient(kubernetesMaster, token));
+    private KubernetesClient buildKubernetesClient(String kubernetesMaster, String accessToken, String caCertificate) {
+        return new RetryKubernetesClient(new DefaultKubernetesClient(kubernetesMaster, accessToken, caCertificate));
     }
 
     @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
@@ -248,6 +248,11 @@ final class HazelcastKubernetesDiscoveryStrategy
         return readFileContents("/var/run/secrets/kubernetes.io/serviceaccount/token");
     }
 
+    @SuppressFBWarnings("DMI_HARDCODED_ABSOLUTE_FILENAME")
+    private static String readCaCertificate() {
+        return readFileContents("/var/run/secrets/kubernetes.io/serviceaccount/ca.crt");
+    }
+
     protected static String readFileContents(String tokenFile) {
         InputStream is = null;
         try {
diff --git a/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactory.java b/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactory.java
index 3545b81..f8fa8ca 100644
--- a/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactory.java
+++ b/src/main/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactory.java
@@ -46,6 +46,7 @@ public class HazelcastKubernetesDiscoveryStrategyFactory
                 KubernetesProperties.RESOLVE_NOT_READY_ADDRESSES,
                 KubernetesProperties.KUBERNETES_MASTER_URL,
                 KubernetesProperties.KUBERNETES_API_TOKEN,
+                KubernetesProperties.KUBERNETES_CA_CERTIFICATE,
                 KubernetesProperties.SERVICE_PORT));
     }
 
diff --git a/src/main/java/com/hazelcast/kubernetes/KubernetesProperties.java b/src/main/java/com/hazelcast/kubernetes/KubernetesProperties.java
index c5ff15d..ef1fa52 100644
--- a/src/main/java/com/hazelcast/kubernetes/KubernetesProperties.java
+++ b/src/main/java/com/hazelcast/kubernetes/KubernetesProperties.java
@@ -104,6 +104,13 @@ public final class KubernetesProperties {
      */
     public static final PropertyDefinition KUBERNETES_API_TOKEN = property("api-token", STRING);
 
+    /**
+     * Configuration key: <tt>ca-certificate</tt>
+     * CA Authority certificate from Kubernetes Master, defaults to reading the certificate from the auto-injected file at:
+     * <tt>/var/run/secrets/kubernetes.io/serviceaccount/ca.crt</tt>
+     */
+    public static final PropertyDefinition KUBERNETES_CA_CERTIFICATE = property("ca-certificate", STRING);
+
     /**
      * <p>Configuration key: <tt>service-port</tt></p>
      * If specified with a value greater than 0, its value defines the endpoint port of the service (overriding the default).


Test patch:
diff --git a/src/test/java/com/hazelcast/kubernetes/DefaultKubernetesClientTest.java b/src/test/java/com/hazelcast/kubernetes/DefaultKubernetesClientTest.java
index 371b041..5ca834b 100644
--- a/src/test/java/com/hazelcast/kubernetes/DefaultKubernetesClientTest.java
+++ b/src/test/java/com/hazelcast/kubernetes/DefaultKubernetesClientTest.java
@@ -42,6 +42,7 @@ public class DefaultKubernetesClientTest {
     private static final String KUBERNETES_MASTER_IP = "localhost";
 
     private static final String TOKEN = "sample-token";
+    private static final String CA_CERTIFICATE = "sample-ca-certificate";
     private static final String NAMESPACE = "sample-namespace";
 
     private static final String SAMPLE_ADDRESS_1 = "192.168.0.25";
@@ -62,7 +63,7 @@ public class DefaultKubernetesClientTest {
     @Before
     public void setUp() {
         String kubernetesMasterUrl = String.format("http://%s:%d", KUBERNETES_MASTER_IP, wireMockRule.port());
-        kubernetesClient = new DefaultKubernetesClient(kubernetesMasterUrl, TOKEN);
+        kubernetesClient = new DefaultKubernetesClient(kubernetesMasterUrl, TOKEN, CA_CERTIFICATE);
     }
 
     @Test
@@ -175,6 +176,20 @@ public class DefaultKubernetesClientTest {
         kubernetesClient.endpoints(NAMESPACE);
     }
 
+    @Test(expected = KubernetesClientException.class)
+    public void nullToken() {
+        // given
+        String kubernetesMasterUrl = String.format("http://%s:%d", KUBERNETES_MASTER_IP, wireMockRule.port());
+        KubernetesClient kubernetesClient = new DefaultKubernetesClient(kubernetesMasterUrl, TOKEN, null);
+
+        stubFor(get(urlEqualTo(String.format("/api/v1/namespaces/%s/pods", NAMESPACE)))
+                .withHeader("Authorization", equalTo(String.format("Bearer %s", TOKEN)))
+                .willReturn(aResponse().withStatus(200).withBody(malformedBody())));
+
+        // when
+        kubernetesClient.endpoints(NAMESPACE);
+    }
+
     /**
      * Real response recorded from the Kubernetes API call "/api/v1/namespaces/{namespace}/pods".
      */
diff --git a/src/test/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactoryTest.java b/src/test/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactoryTest.java
index cc560ea..b9257b7 100644
--- a/src/test/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactoryTest.java
+++ b/src/test/java/com/hazelcast/kubernetes/HazelcastKubernetesDiscoveryStrategyFactoryTest.java
@@ -41,6 +41,7 @@ public class HazelcastKubernetesDiscoveryStrategyFactoryTest {
 
     private static final ILogger LOGGER = new NoLogFactory().getLogger("no");
     private static final String API_TOKEN = "token";
+    private static final String CA_CERTIFICATE = "ca-certificate";
 
     @Mock
     DiscoveryNode discoveryNode;
@@ -73,6 +74,7 @@ public class HazelcastKubernetesDiscoveryStrategyFactoryTest {
     public void createDiscoveryStrategy() {
         HashMap<String, Comparable> properties = new HashMap<String, Comparable>();
         properties.put(KubernetesProperties.KUBERNETES_API_TOKEN.key(), API_TOKEN);
+        properties.put(KubernetesProperties.KUBERNETES_CA_CERTIFICATE.key(), CA_CERTIFICATE);
         properties.put(String.valueOf(KubernetesProperties.SERVICE_PORT), 333);
         HazelcastKubernetesDiscoveryStrategyFactory factory = new HazelcastKubernetesDiscoveryStrategyFactory();
         DiscoveryStrategy strategy = factory.newDiscoveryStrategy(discoveryNode, LOGGER, properties);