Repository: ansible/awx

Issue:
In Kubernetes container groups, user-supplied .metadata.labels is ignored
##### ISSUE TYPE
 - Bug Report

##### SUMMARY

Due to `awx/main/scheduler/kubernetes.py` overriding the `.metadata.labels` of the pods it creates (instead of merging them with the user-supplied `pod_spec_override`), features such as pod anti-affinity between AWX runners cannot work.

##### ENVIRONMENT
* AWX version: 15.0.1 (also present in devel)
* AWX install method: openshift
* Ansible version:  irrelevant
* Operating System: Linux (all versions)
* Web Browser: irrelevant

##### STEPS TO REPRODUCE

1. Create a Kubernetes container group with the below piece of YAML as the pod spec override
1. Run a job out of this instance group

```yaml
apiVersion: v1
kind: Pod
metadata:
  labels:
    deploymentconfig: ansible-runner
  namespace: wwp-test
spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: kubernetes.io/hostname
          labelSelector:
            matchExpressions:
              - key: deploymentconfig
                operator: In
                values:
                  - ansible-runner
  # ...
```

##### EXPECTED RESULTS

The pods run by AWX as part of the container group should contain both the custom labels and the affinity structure.

##### ACTUAL RESULTS

Only the affinity structure shows up in the `Kind: pod` Kubernetes objects, rendering the `podAntiAffinity` clause inoperative (for lack of a label to match on under `metdata`)

##### ADDITIONAL INFORMATION

The cause is the equals sign on [this line](https://github.com/ansible/awx/blob/devel/awx/main/scheduler/kubernetes.py#L132).


Discussion:

Resolving patch:
diff --git a/awx/main/scheduler/kubernetes.py b/awx/main/scheduler/kubernetes.py
--- a/awx/main/scheduler/kubernetes.py
+++ b/awx/main/scheduler/kubernetes.py
@@ -12,6 +12,24 @@
 logger = logging.getLogger('awx.main.scheduler')
 
 
+def deepmerge(a, b):
+    """
+    Merge dict structures and return the result.
+
+    >>> a = {'first': {'all_rows': {'pass': 'dog', 'number': '1'}}}
+    >>> b = {'first': {'all_rows': {'fail': 'cat', 'number': '5'}}}
+    >>> import pprint; pprint.pprint(deepmerge(a, b))
+    {'first': {'all_rows': {'fail': 'cat', 'number': '5', 'pass': 'dog'}}}
+    """
+    if isinstance(a, dict) and isinstance(b, dict):
+        return dict([(k, deepmerge(a.get(k), b.get(k)))
+                     for k in set(a.keys()).union(b.keys())])
+    elif b is None:
+        return a
+    else:
+        return b
+
+
 class PodManager(object):
 
     def __init__(self, task=None):
@@ -128,11 +146,13 @@ def pod_definition(self):
         pod_spec = {**default_pod_spec, **pod_spec_override}
 
         if self.task:
-            pod_spec['metadata']['name'] = self.pod_name
-            pod_spec['metadata']['labels'] = {
-                'ansible-awx': settings.INSTALL_UUID,
-                'ansible-awx-job-id': str(self.task.id)
-            }
+            pod_spec['metadata'] = deepmerge(
+                pod_spec.get('metadata', {}),
+                dict(name=self.pod_name,
+                     labels={
+                         'ansible-awx': settings.INSTALL_UUID,
+                         'ansible-awx-job-id': str(self.task.id)
+                     }))
             pod_spec['spec']['containers'][0]['name'] = self.pod_name
 
         return pod_spec


Test patch: