vagrant@Master:~$ kubectl logs pod  svclb-grafana-03ffa6f9-87fds
error: error from server (NotFound): pods "pod" not found in namespace "default"
vagrant@Master:~$ kubectl logs pod  svclb-grafana-03ffa6f9-87fds -n grafana
error: error from server (NotFound): pods "pod" not found in namespace "grafana"
vagrant@Master:~$ kubectl get pods -A -o wide
NAMESPACE        NAME                                                       READY   STATUS    RESTARTS      AGE   IP                NODE      NOMINATED NODE   READINESS GATES
grafana          grafana-5744c94549-ksw8q                                   1/1     Running   4 (12m ago)   34h   10.42.1.33        worker    <none>           <none>
kube-system      coredns-ff8999cc5-txw22                                    1/1     Running   4 (13m ago)   36h   10.42.0.66        master    <none>           <none>
kube-system      local-path-provisioner-774c6665dc-7njhl                    1/1     Running   5 (13m ago)   36h   10.42.0.57        master    <none>           <none>
kube-system      metrics-server-6f4c6675d5-qjlhq                            1/1     Running   5 (13m ago)   36h   10.42.0.61        master    <none>           <none>
kube-system      svclb-grafana-03ffa6f9-87fds                               0/1     Pending   0             21h   <none>            <none>    <none>           <none>
kube-system      svclb-grafana-03ffa6f9-9qsqq                               0/1     Pending   0             21h   <none>            <none>    <none>           <none>
kube-system      svclb-grafana-03ffa6f9-f5nrh                               0/1     Pending   0             21h   <none>            <none>    <none>           <none>
kube-system      svclb-prometheus-alertmanager-de124004-c6tk2               1/1     Running   4 (12m ago)   34h   10.42.2.30        worker2   <none>           <none>
kube-system      svclb-prometheus-alertmanager-de124004-mtsh7               1/1     Running   4 (13m ago)   34h   10.42.0.62        master    <none>           <none>
kube-system      svclb-prometheus-alertmanager-de124004-sgfr2               1/1     Running   4 (12m ago)   34h   10.42.1.40        worker    <none>           <none>
kube-system      svclb-prometheus-kube-state-metrics-171aeb4a-7fsmj         1/1     Running   4 (12m ago)   34h   10.42.2.32        worker2   <none>           <none>
kube-system      svclb-prometheus-kube-state-metrics-171aeb4a-l4hfh         1/1     Running   4 (12m ago)   34h   10.42.1.36        worker    <none>           <none>
kube-system      svclb-prometheus-kube-state-metrics-171aeb4a-v592b         1/1     Running   4 (13m ago)   34h   10.42.0.59        master    <none>           <none>
kube-system      svclb-prometheus-prometheus-node-exporter-2cf029b2-2jcj4   1/1     Running   4 (13m ago)   34h   10.42.0.63        master    <none>           <none>
kube-system      svclb-prometheus-prometheus-node-exporter-2cf029b2-bbsfj   1/1     Running   4 (12m ago)   34h   10.42.2.31        worker2   <none>           <none>
kube-system      svclb-prometheus-prometheus-node-exporter-2cf029b2-fbht8   1/1     Running   4 (12m ago)   34h   10.42.1.37        worker    <none>           <none>
kube-system      svclb-prometheus-prometheus-pushgateway-aac15de8-c7wht     1/1     Running   4 (13m ago)   34h   10.42.0.65        master    <none>           <none>
kube-system      svclb-prometheus-prometheus-pushgateway-aac15de8-hkn47     1/1     Running   4 (12m ago)   34h   10.42.2.34        worker2   <none>           <none>
kube-system      svclb-prometheus-prometheus-pushgateway-aac15de8-v88w8     1/1     Running   4 (12m ago)   34h   10.42.1.39        worker    <none>           <none>
kube-system      svclb-prometheus-server-544c4cf0-9s99f                     0/1     Pending   0             21h   <none>            <none>    <none>           <none>
kube-system      svclb-prometheus-server-544c4cf0-hdhqr                     0/1     Pending   0             21h   <none>            <none>    <none>           <none>
kube-system      svclb-prometheus-server-544c4cf0-mxw8f                     0/1     Pending   0             21h   <none>            <none>    <none>           <none>
kube-system      svclb-traefik-7bfaae3b-27kj8                               2/2     Running   8 (12m ago)   36h   10.42.1.38        worker    <none>           <none>
kube-system      svclb-traefik-7bfaae3b-95s7m                               2/2     Running   8 (13m ago)   36h   10.42.0.64        master    <none>           <none>
kube-system      svclb-traefik-7bfaae3b-sxsn6                               2/2     Running   8 (12m ago)   36h   10.42.2.33        worker2   <none>           <none>
kube-system      traefik-67bfb46dcb-9q82m                                   1/1     Running   4 (12m ago)   36h   10.42.1.34        worker    <none>           <none>
metallb-system   controller-bb5f47665-grkt9                                 1/1     Running   5 (13m ago)   35h   10.42.0.60        master    <none>           <none>
metallb-system   speaker-cfqmq                                              1/1     Running   8 (11m ago)   35h   192.168.121.181   worker2   <none>           <none>
metallb-system   speaker-h95d4                                              1/1     Running   7 (11m ago)   35h   192.168.121.80    worker    <none>           <none>
metallb-system   speaker-k8xhr                                              1/1     Running   5 (13m ago)   35h   192.168.121.41    master    <none>           <none>
prometheus       prometheus-alertmanager-0                                  1/1     Running   4 (12m ago)   34h   10.42.2.36        worker2   <none>           <none>
prometheus       prometheus-kube-state-metrics-c7cbfcc5d-gbn76              1/1     Running   5 (13m ago)   34h   10.42.0.67        master    <none>           <none>
prometheus       prometheus-prometheus-node-exporter-2bs59                  1/1     Running   2 (12m ago)   21h   10.42.1.35        worker    <none>           <none>
prometheus       prometheus-prometheus-node-exporter-dqp64                  1/1     Running   2 (13m ago)   21h   10.42.0.58        master    <none>           <none>
prometheus       prometheus-prometheus-node-exporter-z7vcl                  1/1     Running   2 (12m ago)   21h   10.42.2.35        worker2   <none>           <none>
prometheus       prometheus-prometheus-pushgateway-79c99bd4b8-mrzqv         1/1     Running   4 (13m ago)   34h   10.42.0.69        master    <none>           <none>
prometheus       prometheus-server-564448f5dc-5svvd                         2/2     Running   8 (13m ago)   34h   10.42.0.68        master    <none>           <none>
vagrant@Master:~$ kubectl logs svclb-grafana-03ffa6f9-87fds -n kube-system
vagrant@Master:~$ kubectl logs svclb-grafana-03ffa6f9-87fds -n kube-system
vagrant@Master:~$ kubectl describe pod svclb-grafana-03ffa6f9-87fds -n kube-system
Name:                 svclb-grafana-03ffa6f9-87fds
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      svclb
Node:                 <none>
Labels:               app=svclb-grafana-03ffa6f9
                      controller-revision-hash=5757df9859
                      pod-template-generation=1
                      svccontroller.k3s.cattle.io/svcname=grafana
                      svccontroller.k3s.cattle.io/svcnamespace=grafana
Annotations:          <none>
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        DaemonSet/svclb-grafana-03ffa6f9
Containers:
  lb-tcp-80:
    Image:      rancher/klipper-lb:v0.4.13
    Port:       80/TCP
    Host Port:  80/TCP
    Environment:
      SRC_PORT:    80
      SRC_RANGES:  0.0.0.0/0
      DEST_PROTO:  TCP
      DEST_PORT:   80
      DEST_IPS:    10.43.171.31
    Mounts:        <none>
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:         <none>
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly op=Exists
                 node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                 node-role.kubernetes.io/master:NoSchedule op=Exists
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  19h (x22 over 21h)   default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  9m48s (x2 over 14m)  default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 Preemption is not helpful for scheduling.
vagrant@Master:~$ kubectl describe pod svclb-grafana-03ffa6f9-87fds -n kube-system
Name:                 svclb-grafana-03ffa6f9-87fds
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      svclb
Node:                 <none>
Labels:               app=svclb-grafana-03ffa6f9
                      controller-revision-hash=5757df9859
                      pod-template-generation=1
                      svccontroller.k3s.cattle.io/svcname=grafana
                      svccontroller.k3s.cattle.io/svcnamespace=grafana
Annotations:          <none>
Status:               Pending
IP:                   
IPs:                  <none>
Controlled By:        DaemonSet/svclb-grafana-03ffa6f9
Containers:
  lb-tcp-80:
    Image:      rancher/klipper-lb:v0.4.13
    Port:       80/TCP
    Host Port:  80/TCP
    Environment:
      SRC_PORT:    80
      SRC_RANGES:  0.0.0.0/0
      DEST_PROTO:  TCP
      DEST_PORT:   80
      DEST_IPS:    10.43.171.31
    Mounts:        <none>
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:         <none>
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     CriticalAddonsOnly op=Exists
                 node-role.kubernetes.io/control-plane:NoSchedule op=Exists
                 node-role.kubernetes.io/master:NoSchedule op=Exists
                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                 node.kubernetes.io/not-ready:NoExecute op=Exists
                 node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                 node.kubernetes.io/unreachable:NoExecute op=Exists
                 node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  19h (x22 over 21h)  default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  10m (x2 over 15m)   default-scheduler  0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 node(s) didn't satisfy plugin(s) [NodeAffinity]. preemption: 0/3 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 2 Preemption is not helpful for scheduling.
vagrant@Master:~$ kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml | grep -A10 affinity
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - worker
  automountServiceAccountToken: false
  containers:
vagrant@Master:~$ kubectl get pod svclb-grafana-03ffa6f9-87fds -n kube-system -o yaml | grep -A10 hostPort
      hostPort: 80
      name: lb-tcp-80
      protocol: TCP
    resources: {}
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  dnsPolicy: ClusterFirst
vagrant@Master:~$ sudo netstat -tuln | grep ':80'
vagrant@Master:~$