Repository: prequel-dev/cre (go)

Issue:
Istio Ambient Troubleshooting Rules
### Turn a manual troubleshooting guide into a library of [CREs](https://docs.prequel.dev/cres/introduction)

We have a technical guide that walks through a specific set of reliability issues. Your task is to break it down into a structured set of CREs that can be shared with the community and used by `preq` to automate problem detection.

This a great opportunity to get hands-on experience with increasingly popular technologies.  

### 📌 What you’ll do

- Read the [guide](https://github.com/istio/istio/wiki/Troubleshooting-Istio-Ambient)
- Extract and document multiple CREs based on the key failure modes, signals, and remediations described.
- Submit them in the [Prequel CRE format](https://github.com/prequel-dev/cre#cre-format) following the structure of this repo.  

### ✅ What we’re looking for

- At least 10 high-quality CREs 


### 🏆 Award

**$325** if the submission hits all criteria

### ⚙️ Process
* We'll leave this bounty unassigned for 5 days [June 15 11:59 pm ET].  *We encourage anyone interested to comment /attempt #81 on the issue*
* We will then assign the bounty to a someone based on 
   * quality of plan 
   * past contributions
** The assignee will have 5 days to exclusively pursue the bounty. If the work is not completed, the bounty may be reassigned **


✅ **You must:**

1.  Reproduce the failure scenarios
2.  Share a minimal, working reproductions (e.g., Helm, Docker Compose, etc.)
3.  Write unique [CRE](https://docs.prequel.dev/cres/introduction)-format detection rules for [`preq`](https://github.com/prequel-dev/preq)
4. [Install](https://docs.prequel.dev/install) `preq` and run the rules

📦 **Deliverables:**

*  Reproduction setup and clear README
    *  create a private repo and invite the maintainers (@tonymeehan @lyndon-prequel)
*  Link to your rules in seperate [CRE playgrounds](https://play.prequel.dev/)
*  PR containing 
     * the new CRE rules
     * example logs in test.log files
     * updated tags.yaml and category.yaml files 
     * updated data-sources.yaml file
*  Demo video of 
   *  your reproductions
   *  running the rules

### 📅 Deadline

Will be assigned by June 15
Submit your pull request by **Submit by June 20 11:59 pm ET**

Questions? Open a discussion or find us on [slack](https://inviter.co/prequel). Happy hunting!

/bounty 325

Pull request description:
Add Istio Ambient critical upstream failure detection rule
This PR adds a new CRE rule to detect Ambient CNI sandbox‐creation failures in Istio Ambient mode. The rule catches high‐severity failure mode:

`No ztunnel connection`: CNI plugin cannot contact the node‐level ztunnel agent

These failure leave pods stuck permanently in `ContainerCreating` (or `Pending`), preventing any workloads from starting in Istion Ambient mesh.

Part of #81 
Closes #81 

## Test Environment
 **Live CRE link:** [CRE Playground Link](https://play.prequel.dev/?data=EwBmFYFoQNk4CMAVBMBcAWYaQGYBaABMQMoAuAhgE5kCmAJsU8y8%2BdXYwMYD2AdpQCWfWlUKCAzmUE9IAByo8AHgE8AUKAjQ4iFOiyYYRYgGlBAG3PCA5qzstyPOXJuFeAisNHipM%2BYtUNMChYeGRUTGwMIyYzS1d7RMdnV3chETEJCj56ACNlSAAzT3MgrVDdCIxcNAQQY0IAVT4AC1oKczIWlUS7ACV2%2Bi8JCUIFHlzaQmKLBjRCAHFaMkIAIhayMjk0AHoduoA6YAwMA5ADhHBa8DAEHbaOroAvHapBlVX5oY7CMi45QiHY6nc6Xa63ebuERcMiQ-jQ6T8QhvQoAVwkDDKIR04X0NWAMVIXDa9FR5gYvVYJFRXC4tBGaMsPQoI0E1hEjHotGKZLIOyyOXySiKJV%2BPB8iMgXIAtrI0opzPJzNlaFjtGE9Jh8YTCAAxEoMADCbwodAACjx6CRsvQAELKJj62aMMjirgmuhjS2EAV5ZTzKhyLiEUSKKhwrmEAC8TT4AGs%2BDwAO58QhciTBmMzcku8UYsiogEiMhJnhUOPTMs%2Bm1CtYADhAFDr4C4uBAAHZchBCgg67kAJztjC0XIgfu4NvtYCFQowXAIYeFWgIBCzhC0GB1%2Bj0OtLii4dsIXLtg8T8AYOufMbmVHWYS-FRyWhR1aSSVcPiCVaEPgUaXP19fDlT9v2zCkAAoKG3ABKeY3z8D9BDcaV6AAQW3aYDVzNx%2BEoGEf0tKYAEkgMIQ0ADkiMIChrFoAR5lRX9cnJMUxnRFoyMokMADc6JWcCpFNdEcMjG4QFggjCCeAs%2BBEcwcNk2gYRkPg1QqXEtTQYB2waJ0c2NdpzUta0cntJRHSw1j3UMqY5G9X0hQDIMQyoMMIymGNmgTZNU3TTNMOdVj80LH9llLctK0yGsHVWBBcGbXB6GAUcQAwft%2B1oOsKAocAECSmAR1ybL20KYBaFwWhwH7QpaVbVKuAXWhGzqXBgDrDAW1wDAKFoZc6ivOQbzvVMyEfAD4OAr8fz-cagMgRDQMsyCYLgubEOQtCMLA7C0gofDE0jEjEU4qiaL4himJY102IkDiKKo2heIEQgBMoAtRl4USQHE%2BZEykmS5IUhEVLUnFNWqTB6gs50DNNWgLStG0zOhnMrI9Wz7OipQnODUMy3c6NY28lM03pfztqC5YQuLcKK0KKsHJittqsHfs0voJtcDZigOwQCgMC5GA8G6qDaG6m52zrGAYAy8q2oQYAuHAcB6HAErKvafndxABABqG%2B9RqfF8JvmkDpv-Y21rNinlvoCSTfWrgUPQxgKeu3b9sIwgjpkE7qNo%2BjCEYihmKma65HYv3Hr4l7BPekSpjEiS-ukxjAahJTEVUzRsQ1KoamV3SsNhozEdMh1iD0il3fRr1GEZ7HkWcvHwwTwmvMTEm-MJt28yposwrLOmGaxtYYAHArVeKBASs7Ch22AbT6CwMBYEKXL1-odsuEKLl6Gl1BihgHKQHoYpO3a9twGAftEDrBB%2Bz128DbGy33zN38LcA9%2BpptqC7dWj-DaLsAqo3drhPaKwDrEVIvdf250g6XTDuKCOt0o5PX4nHYSn1E7fWTuKVOil5IZ2UvwUGed9AYBwHWIuMN0YIxMnaCuepLI1xsnXasgp-RN1xq5fGbdPLxk7r5MmPdWF9wLAPEsQ9IqcL9OZVYMA8i7kPAracpUKqKyXDrCgMBaR1nbLQdsIBKpHhuBecc9AH77ySlgE8WB%2BzgGbGLCg-YYDtnbE-YaD4jbfwQh-Gab9-G-yWv-e2VskJO02q7cRCk8JQK9j7cUcCzqB2DqHViqC7pcWjs9V6QkPpeyTr9AhANaDEPhJnFSQA&rule=E4VwNgpgzgXAUAAgQWgQWwgFwIYBNs7xLEIDWAlgHa4wIAOwEAjiBGIiQuTQgEYDSAFkoAtNABkAHgAVcAFQCKADgBuACQBWUAJoBVSQDVeHEgAtsUU7QXA5AVgCeAcVz8A6hAAat3pW11xKDUAYwA5EwRgxiJObloAYQAlAFFkACYABjS7ZAyARgyANgikKAgVCGByTAdaPJKETGrIWgAiAEE0XnIISkwEeNCASQQAZWxqXgB7SQHGAnIpygQAMWxyMBBGVobggggAcyngWoRW8igmqeRsLp6%2B5ExgKZBeSEspqabKA53OBGwIEwpmOtGkjBYbAauGgUXIdCulFoAB8GkgACJYCDBTBQBAAd1MvUaRIQQ0uiwGw3omwOVAQADN1mA8Zgpggyv0QHQAfQprgAOR4yhYfHHUgcia4aaSNFcZadbq9fpofkQAB0CDkYsiUzQquWzy%2BkUBZTx2GicrleU1ACpbaF2QAvTAgSgisC693YxH2hAACkGIz27uNwSWOBxJIgCEoauQkAqnpdbo9AIOyoAlJqELtTRAUXLUCnvZ70dgIAbRlguHi3VBgkTcOAILgEMcEG6idgwMCHO3lsCY3GYUWqSM6LT6SG4-1cOQe9GECW07pKORZlApsFSDWAAYAehUFoPoEoB4uV2QwXXB5v5HVW53e4a5DQdGwOML-yQqGk-LxS4QB3eUBgjdYRWAeJ5m%2BA522ABBpF6ecfjHNxxTAKY8DxGdjUuC1MAAGgQSA8CoOC2QBOhJ3IPZEQQXApnxPo3wgBo0GqcgDgWJZvx-IYAFkBOSdEhnaORkhif5UAMSpyAZfsVzYBBy0rJZq36bteysBA91IV4fU9ZBlkvRZkCgBxLkrBAM36OgAJQT1sGogBeJSwBfH8UDJSgoDoH1l1dUsSKmA48QZDtw29HFyBUap%2B0qZ5gFgXMvKQPSDJxIyTIpa4LKstAQrChAAB53OQezcAAPhfOUUniAB5AxkkSbQpM4P8CEbQLU2UyizwHAEwE9EdoHa-4Mt4QyUByq98swayP0wbr8DU3ya3clAKq60wEAFABvVo-OxVoYEOhb3zAfZTsO47ghu1pRurSAcWOG6AF9Ps%2BgVPJ-VBMUgBaAWoBBGCiCB9iXbAGQZH1Wz5XAoCtLzpBSZrQjkIYGtCcbiFQASlmqDtySucdgbbTbsAzPp0CwKpgiRrzUHaSBgH6JZQYgfz9jbIMaRAOlliZDYtmgUCoqeKYwAqq6RSKxnOBwMLcd-WsrjHe8x1uJU%2BgaRhYcYShgjGsdTEwTA6FgA8DzpYFXnVcM0AvXLnauA98XICgDzkZ5XneEEvnI5ASbMxV7kwABiBtegtMzKuQYWWUea4z2QD3gQT5lW2T68YIgCr%2BXMqUZQaZyaLoxZfJV7zKFuAs1bMzW0oQCpksrtoqpcm00kKdUMj%2BYhQBaBpOWrj3qCY2hCgyBX-nKZVq9KF5gGN2h9KmrL1RhBsqim4B1UquU0B2xfvMYDNJDaNYNlbaCIYW-9cHGagACEZgHpnOYvto4x64KopFDFJYH9OAim4gtWgABtAAunAIAA)

## Reproducer: Ambient CNI Sandbox Creation Failure

Follow these steps to reproduce the “Failed to create pod sandbox” error due to no ztunnel connection in Istio Ambient mode.

### 1. Create the test pod

```bash
~/git/istio$ kubectl run sandbox-fail \
  --image=busybox \
  --restart=Never \
  -- sleep 3600
```

### 2. Verify Pod status

```bash
~/git/istio$ kubectl get pod sandbox-fail
```
You should see output similar to:

```log
NAME           READY   STATUS              RESTARTS   AGE
sandbox-fail   0/1     ContainerCreating   0          10s
```

### 3. Inspect failure events with real timestamps

```bash
~/git/istio$ kubectl get events \
  --field-selector involvedObject.name=sandbox-fail,involvedObject.kind=Pod \
  --sort-by='.metadata.creationTimestamp' \
  -o custom-columns='TIME:.lastTimestamp,REASON:.reason,MESSAGE:.message'
  ```
  Expected output:

```log

TIME                   REASON                   MESSAGE
2025-06-21T16:42:03Z   Created                  Created container istio-proxy
2025-06-21T16:42:03Z   Started                  Started container istio-proxy
2025-06-21T16:42:46Z   Killing                  Stopping container istio-proxy
2025-06-21T16:42:46Z   Killing                  Stopping container sandbox-fail
2025-06-21T16:43:10Z   Unhealthy                Readiness probe failed: Get "http://10.244.0.15:15021/healthz/ready": dial tcp 10.244.0.15:15021: connect: connection refused
2025-06-21T16:43:26Z   Scheduled                Successfully assigned default/sandbox-fail to istio-demo-control-plane
2025-06-21T16:43:26Z   FailedCreatePodSandBox   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "80a85c307b25f18b974eb09330ea2ff6314efe11f61e68dd8fea371b73733548": plugin type="istio-cni" name="istio-cni" failed (add): istio-cni cmdAdd failed to contact node Istio CNI agent: unable to push CNI event (status code 500): no ztunnel connection
```

## DEMO

https://github.com/user-attachments/assets/7f7e118a-0883-4d78-b8d3-d70126f8e4ea




Patch:
diff --git a/rules/cre-2025-106/ambient-cni-sandbox-creation-failure.yaml b/rules/cre-2025-106/ambient-cni-sandbox-creation-failure.yaml
new file mode 100644
index 0000000..a6232f5
--- /dev/null
+++ b/rules/cre-2025-106/ambient-cni-sandbox-creation-failure.yaml
@@ -0,0 +1,53 @@
+rules:
+  - metadata:
+      kind: prequel
+      id: bK4nZmLxPdTQ8vHjsYUxVb
+      hash: QrT5yGdKWeXrTbnYpLsHcN
+    cre:
+      id: CRE-2025-0106
+      severity: 1
+      title: "Ambient CNI Sandbox Creation Failure"
+      category: "istio-ambient-troubleshooting"
+      author: Prequel
+      description: |
+        Detects when the Istio CNI plugin fails to set up a pod's network sandbox
+        in Ambient mode. Two common root causes are:
+        
+        1. **No ztunnel connection** (CNI cannot contact the node-level ztunnel agent).  
+      cause: |
+        - ztunnel DaemonSet is unscheduled or unhealthy on the node
+        - CNI plugin cannot dial the ztunnel Unix socket `/var/run/istio-cni/cni.sock`
+      impact: |
+        - Pods stuck in ContainerCreating or Pending
+        - Workloads cannot start, leading to application downtime
+      mitigation: |
+        IMMEDIATE:
+        - Verify ztunnel DaemonSet health: `kubectl -n istio-system get pods -l app=ztunnel`
+        - Inspect ztunnel logs for connectivity errors: 
+          `kubectl -n istio-system logs <ztunnel-pod>`
+
+        RECOVERY:
+        - Patch ztunnel to run on all nodes:
+          `kubectl -n istio-system patch daemonset ztunnel --patch '{"spec":{"template":{"spec":{"nodeSelector":{}}}}}'`
+        - Delete and recreate the affected pods
+        
+        PREVENTION:
+        - Monitor Istio CNI and ztunnel agent metrics
+        - Alert on repeated CNI plugin failures in control-plane logs
+      tags:
+        - istio
+        - cni
+        - ambient
+      references:
+        - https://github.com/istio/istio/wiki/Troubleshooting-Istio-Ambient#scenario-pod-fails-to-run-with-failed-to-create-pod-sandbox
+      applications:
+        - name: istio-cni
+          version: ">=1.26.0"
+    rule:
+      set:
+        window: 60s
+        event:
+          source: cre.log.ambient
+        match:
+          - regex: "FailedCreatePodSandBox"
+          - regex: "no ztunnel connection"
diff --git a/rules/tags/categories.yaml b/rules/tags/categories.yaml
index 62e9ef2..469cb1d 100644
--- a/rules/tags/categories.yaml
+++ b/rules/tags/categories.yaml
@@ -197,3 +197,4 @@ categories:
       Problems related to troubleshooting Istio's Ambient service mesh mode,
       including CNI sandbox creation failures, ztunnel connectivity issues,
       traffic capture errors, and waypoint configuration problems.
+


Test patch:
diff --git a/rules/cre-2025-106/test.log b/rules/cre-2025-106/test.log
new file mode 100644
index 0000000..9035717
--- /dev/null
+++ b/rules/cre-2025-106/test.log
@@ -0,0 +1,9 @@
+2025-06-21T16:42:03Z   Started                  Started container istio-proxy
+2025-06-21T16:42:46Z   Killing                  Stopping container istio-proxy
+2025-06-21T16:42:46Z   Killing                  Stopping container sandbox-fail
+2025-06-21T16:43:26Z   Scheduled                Successfully assigned default/sandbox-fail to istio-demo-control-plane
+2025-06-21T16:43:26Z   FailedCreatePodSandBox   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "80a85c307b25f18b974eb09330ea2ff6314efe11f61e68dd8fea371b73733548": plugin type="istio-cni" name="istio-cni" failed (add): istio-cni cmdAdd failed to contact node Istio CNI agent: unable to push CNI event (status code 500): no ztunnel connection
+2025-06-21T16:43:27Z   FailedCreatePodSandBox   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "13853d2b00499e8aa51d26ebbaa7f2e3e59fccc304c14e0a1032845c34aee110": plugin type="istio-cni" name="istio-cni" failed (add): istio-cni cmdAdd failed to contact node Istio CNI agent: unable to push CNI event (status code 500): no ztunnel connection
+2025-06-21T16:43:40Z   FailedCreatePodSandBox   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "309f97949da8394a071a4de6034aade4a5078669ee32812c55d57fe5eaa48f01": plugin type="istio-cni" name="istio-cni" failed (add): istio-cni cmdAdd failed to contact node Istio CNI agent: unable to push CNI event (status code 500): no ztunnel connection
+2025-06-21T16:43:55Z   FailedCreatePodSandBox   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6b96ed5fa17f7ba7227d420206f51f5d7cfded6616fa6a50dfa7b84752921819": plugin type="istio-cni" name="istio-cni" failed (add): istio-cni cmdAdd failed to contact node Istio CNI agent: unable to push CNI event (status code 500): no ztunnel connection
+2025-06-21T16:44:08Z   FailedCreatePodSandBox   Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6db8f71122ff23e2cfe01a6cc87e70e51b504893d19d6d24273429585e4a9677": plugin type="istio-cni" name="istio-cni" failed (add): istio-cni cmdAdd failed to contact node Istio CNI agent: unable to push CNI event (status code 500): no ztunnel connection
\ No newline at end of file