Skill: royhk920/debug-detective (version 1.0.0)

--- name: debug-detective description: > Activate this skill whenever a user needs help debugging, diagnosing, or profiling issues in their application. Covers systematic debugging methodology, git bisect, Chrome DevTools deep dive, Node.js and Python debugger usage, system-level debugging with strace and tcpdump, database query debugging with EXPLAIN ANALYZE, network debugging with curl and DNS tools, memory leak detection, CPU and performance profiling with flame graphs, structured logging and distributed tracing with OpenTelemetry, and production debugging strategies. Provides step-by-step workflows and a comprehensive pitfalls reference. license: MIT compatibility: "Openclaw" metadata: version: "1.0.0" author: "OpenClaw" category: "debugging" --- # Debug Detective — Systematic Debugging Methodology Find and fix bugs efficiently across the full stack using structured investigation techniques. ## 1. Debugging Mindset ### 1.1 The scientific method for debugging ``` 1. OBSERVE — What exactly is happening? (symptoms, error messages, logs) 2. HYPOTHESIZE — What could cause this? (list 3+ possibilities) 3. PREDICT — If hypothesis X is correct, then Y should be true 4. TEST — Design the smallest experiment to test the prediction 5. ANALYZE — Did the test confirm or refute the hypothesis? 6. REPEAT — If refuted, move to next hypothesis; if confirmed, fix and verify ``` ### 1.2 Key debugging principles - **The bug is never where you think it is.** Widen your search radius before going deep. - **Reproduce first, fix second.** A bug you can't reproduce is a bug you can't verify as fixed. - **Change one thing at a time.** Multiple simultaneous changes make it impossible to identify the fix. - **Trust nothing.** Verify assumptions — check that the code you're reading is the code that's running. - **Read the error message.** Fully. Including the stack trace. Including the "caused by" chain. ### 1.3 Cognitive biases that hinder debugging | Bias | How it hurts | Counter-strategy | |------|-------------|------------------| | **Confirmation bias** | You look for evidence supporting your theory, ignore contradicting evidence | Actively try to disprove your hypothesis | | **Anchoring** | First theory dominates even when evidence points elsewhere | Write down 3+ hypotheses before investigating any | | **Recency bias** | "I just changed X, so X must be the problem" | Check git log — the bug might predate your change | | **Availability bias** | "Last time it was a race condition, so it must be again" | Consider all categories: data, logic, timing, config, environment | | **Sunk cost** | "I've spent 2 hours on this theory, it must be right" | Set a timebox: 30 min per hypothesis, then move on | ### 1.4 Rubber duck debugging Explain the problem out loud (to a duck, a colleague, or a text file): 1. State what the code is supposed to do 2. Walk through the code line by line, explaining each step 3. The act of articulating often reveals the gap between expectation and reality ### 1.5 Feynman technique 1. Write the bug description as if explaining to a non-programmer 2. Identify gaps in your explanation — those are gaps in your understanding 3. Go back to the code to fill those gaps 4. Simplify your explanation further ## 2. Systematic Debugging Workflow ### 2.1 The six-step process ``` ┌─────────────┐ │ 1. REPRODUCE │ ← Can you trigger the bug reliably? └──────┬──────┘ ▼ ┌─────────────┐ │ 2. ISOLATE │ ← Narrow down: which component, input, or path? └──────┬──────┘ ▼ ┌─────────────┐ │ 3. IDENTIFY │ ← Root cause found └──────┬──────┘ ▼ ┌─────────────┐ │ 4. FIX │ ← Minimal, targeted change └──────┬──────┘ ▼ ┌─────────────┐ │ 5. VERIFY │ ← Bug no longer reproduces; no regressions └──────┬──────┘ ▼ ┌─────────────┐ │ 6. PREVENT │ ← Add test, monitoring, or documentation └─────────────┘ ``` ### 2.2 Reproducing the bug **Minimal reproduction checklist:** 1. Start from a clean state (fresh install, empty database, incognito browser) 2. List exact steps to trigger the bug 3. Note the environment: OS, runtime version, browser, config 4. Strip away unrelated code until the bug is isolated 5. If intermittent: identify the timing/concurrency pattern ```bash # Create a minimal reproduction project mkdir bug-repro && cd bug-repro npm init -y # Add only the minimum dependencies needed to demonstrate the bug npm install problematic-library@1.2.3 # Write the smallest possible script that triggers the issue ``` ### 2.3 Binary search debugging When you don't know where the bug is, bisect: **Code bisection:** ``` // Add a return/exit at the midpoint of the suspect code // If the bug disappears → bug is after the midpoint // If the bug persists → bug is before the midpoint // Repeat on the narrowed half ``` **Data bisection:** ```bash # If a large input causes the bug, split it in half head -n 500 input.csv > first_half.csv tail -n 500 input.csv > second_half.csv # Test each half — which one triggers the bug? ``` **Config bisection:** ```bash # Comment out half the config, test # Narrow down which config option causes the issue ``` ### 2.4 Reading stack traces ``` Error: Cannot read properties of undefined (reading 'email') at getUserEmail (src/services/user.ts:42:18) ← WHERE it crashed at processOrder (src/services/order.ts:87:24) ← WHO called it at OrderController.create (src/controllers/order.ts:23:5) ← Entry point at Layer.handle (node_modules/express/lib/router/layer.js:95:5) ``` **Read bottom-up:** The bottom shows where the call originated. The top shows where it failed. The line `src/services/user.ts:42` is where to look, but the **cause** might be in `order.ts:87` (passing undefined). ## 3. Git Bisect ### 3.1 Manual bisect ```bash # Start bisecting git bisect start # Mark current (broken) commit as bad git bisect bad # Mark a known-good commit (e.g., last release tag) git bisect good v2.0.0 # Git checks out the midpoint — test it # If this commit is broken: git bisect bad # If this commit works: git bisect good # Repeat until git identifies the first bad commit # Git outputs: "abc1234 is the first bad commit" # Done — reset git bisect reset ``` ### 3.2 Automated bisect ```bash # Automated: provide a test script that exits 0 (good) or 1 (bad) git bisect start HEAD v2.0.0 git bisect run npm test # Or with a custom script git bisect run bash -c ' npm run build 2>/dev/null && \ node -e " const { buggyFunction } = require(\"./dist\"); const result = buggyFunction(\"test-input\"); process.exit(result === expected ? 0 : 1); " ' # Reset when done git bisect reset ``` ### 3.3 Bisect with skip ```bash # If a commit can't be tested (e.g., build broken for unrelated reason) git bisect skip # Skip a range of untestable commits git bisect skip v2.0.1..v2.0.5 ``` ## 4. Frontend Debugging ### 4.1 Chrome DevTools — Console power features ```js // $0 — reference to currently selected element in Elements panel $0.textContent // $$() — querySelectorAll shortcut $$('button.primary').length // copy() — copy any value to clipboard copy(JSON.stringify(data, null, 2)) // monitor() — log all calls to a function monitor(fetch) // unmonitor(fetch) to stop // monitorEvents() — log all events on an element monitorEvents($0, 'click') // unmonitorEvents($0) to stop // queryObjects() — find all instances of a constructor queryObjects(Promise) // Find all live Promises // table() — display array/object as table console.table(users, ['name', 'email', 'role']) // time/timeEnd — measure execution time console.time('render') renderComponent() console.timeEnd('render') // render: 42.3ms // group — organize related logs console.group('API Request') console.log('URL:', url) console.log('Method:', method) console.log('Body:', body) console.groupEnd() // assert — log only when condition fails console.assert(user.id, 'User ID is missing', user) ``` ### 4.2 Sources panel — Advanced breakpoints | Breakpoint type | How to set | Use case | |----------------|-----------|----------| | Line breakpoint | Click line number | Stop at specific line | | Conditional | Right-click line → "Add conditional" | Stop only when condition is true | | Logpoint | Right-click → "Add logpoint" | Log without modifying code | | DOM breakpoint | Elements panel → right-click → "Break on" | Stop when DOM changes | | XHR breakpoint | Sources → XHR Breakpoints → add URL pattern | Stop on matching fetch/XHR | | Event listener | Sources → Event Listener Breakpoints | Stop on click, keypress, etc. | | Exception | Sources → pause icon → "Pause on exceptions" | Stop on any thrown error | ### 4.3 Performance panel — Finding slow code ``` 1. Click Record (or Ctrl+E) 2. Perform the slow action in the app 3. Click Stop 4. Analyze the flame chart: - Wide bars = slow functions - Look for "Long Task" markers (>50ms) - Check "Bottom-Up" tab for aggregate time per function - Check "Call Tree" for the hot path ``` ### 4.4 Memory panel — Finding leaks ``` 1. Take Heap Snapshot (baseline) 2. Perform the action suspected of leaking 3. Take another Heap Snapshot 4. Select Snapshot 2, change view to "Comparison" 5. Sort by "# Delta" — positive deltas are new allocations 6. Look for: - Detached DOM trees (elements removed from page but still referenced) - Growing arrays or maps - Event listeners not cleaned up ``` ### 4.5 CSS debugging techniques ```css /* Outline all elements to see layout issues */ * { outline: 1px solid red !important; } /* More detailed — color by nesting depth */ * { outline: 1px solid rgba(255, 0, 0, 0.3) !important; } * * { outline: 1px solid rgba(0, 255, 0, 0.3) !important; } * * * { outline: 1px solid rgba(0, 0, 255, 0.3) !important; } /* Debug z-index stacking */ * { position: relative; } *::after { content: attr(class); position: absolute; top: 0; left: 0; font-size: 10px; background: yellow; z-index: 99999; } ``` ### 4.6 React DevTools profiler ``` 1. Open React DevTools → Profiler tab 2. Click Record 3. Interact with the app 4. Click Stop 5. Analyze: - Flame chart shows component render times - Ranked chart shows slowest components - "Why did this render?" shows trigger reasons - Look for unnecessary re-renders (grey = didn't render) ``` ## 5. Node.js / JavaScript Debugging ### 5.1 Inspect flag ```bash # Start with debugger node --inspect src/server.js # Break on first line node --inspect-brk src/server.js # Then open chrome://inspect in Chrome and click "inspect" ``` ### 5.2 VS Code launch.json ```json { "version": "0.2.0", "configurations": [ { "name": "Debug Server", "type": "node", "request": "launch", "program": "${workspaceFolder}/src/server.ts", "runtimeExecutable": "tsx", "console": "integratedTerminal", "env": { "NODE_ENV": "development" } }, { "name": "Debug Tests", "type": "node", "request": "launch", "program": "${workspaceFolder}/node_modules/.bin/vitest", "args": ["run", "--reporter=verbose", "${file}"], "console": "integratedTerminal" }, { "name": "Attach to Process", "type": "node", "request": "attach", "port": 9229, "restart": true } ] } ``` ### 5.3 Memory leak hunting in Node.js ```bash # Take heap snapshots programmatically node --expose-gc -e " const v8 = require('v8'); const fs = require('fs'); global.gc(); // Force GC before snapshot const snapshot = v8.writeHeapSnapshot(); console.log('Snapshot written to:', snapshot); " # Monitor memory usage over time node -e " setInterval(() => { const used = process.memoryUsage(); console.log( 'RSS:', (used.rss / 1024 / 1024).toFixed(1), 'MB', 'Heap:', (used.heapUsed / 1024 / 1024).toFixed(1), 'MB' ); }, 5000); " ``` ### 5.4 Debugging async code ```ts // Enable async stack traces (default in Node 16+) // Errors will show the full async call chain // Common async debugging pattern: add context to errors async function processOrder(orderId: string) { try { const order = await fetchOrder(orderId); const payment = await chargePayment(order); return await fulfillOrder(order, payment); } catch (error) { // Wrap with context — don't lose the original stack throw new Error(`Failed to process order ${orderId}`, { cause: error }); } } ``` ### 5.5 Why is Node.js not exiting? ```bash # Find what's keeping Node alive npm install why-is-node-running ``` ```ts import why from "why-is-node-running"; // After your work is done, if the process doesn't exit: setTimeout(() => { why(); // Prints active handles keeping the process alive }, 5000); ``` ## 6. Python Debugging ### 6.1 Built-in debugger ```python # Insert breakpoint anywhere in code def process_data(items): result = [] for item in items: breakpoint() # Drops into pdb (or ipdb if installed) transformed = transform(item) result.append(transformed) return result # pdb commands: # n — next line (step over) # s — step into function # c — continue to next breakpoint # p expr — print expression # pp expr — pretty print # l — show current code location # w — show call stack # u/d — move up/down the call stack # b 42 — set breakpoint at line 42 # cl — clear all breakpoints # q — quit debugger ``` ### 6.2 ipdb (enhanced debugger) ```bash pip install ipdb ``` ```python # Use ipdb instead of pdb for better UX (tab completion, syntax highlighting) import ipdb; ipdb.set_trace() # Or set as default debugger # In ~/.bashrc or environment: # export PYTHONBREAKPOINT=ipdb.set_trace ``` ### 6.3 py-spy — Production profiling ```bash pip install py-spy # Profile a running process (no restart needed!) py-spy top --pid 12345 # Generate flame graph py-spy record -o flamegraph.svg --pid 12345 # Profile a script py-spy record -o flamegraph.svg -- python my_script.py # Dump current stack traces py-spy dump --pid 12345 ``` ### 6.4 Memory profiling ```bash pip install memory-profiler ``` ```python from memory_profiler import profile @profile def memory_intensive(): data = [i ** 2 for i in range(1_000_000)] # Watch memory spike here filtered = [x for x in data if x % 2 == 0] return len(filtered) # Output shows per-line memory usage: # Line # Mem usage Increment Line Contents # 4 45.2 MiB 0.0 MiB @profile # 5 45.2 MiB 0.0 MiB def memory_intensive(): # 6 83.5 MiB 38.3 MiB data = [i ** 2 for ...] # 7 99.1 MiB 15.6 MiB filtered = [x for ...] ``` ### 6.5 tracemalloc — Built-in memory tracking ```python import tracemalloc tracemalloc.start() # ... run your code ... snapshot = tracemalloc.take_snapshot() top_stats = snapshot.statistics("lineno") print("Top 10 memory allocations:") for stat in top_stats[:10]: print(stat) ``` ## 7. System-Level Debugging ### 7.1 strace — Trace system calls ```bash # Trace a running process strace -p 12345 # Trace a command from start strace -f node server.js # -f follows child processes # Only trace specific calls strace -e trace=open,read,write node server.js # Trace network calls only strace -e trace=network node server.js # Count call statistics strace -c node server.js # Output to file with timestamps strace -tt -o trace.log node server.js ``` **Common findings:** ``` open("/etc/resolv.conf", O_RDONLY) = -1 ENOENT ← DNS config missing connect(3, {AF_INET, 10.0.0.5:5432}, 16) = -1 ETIMEDOUT ← DB unreachable write(1, "Error: ENOSPC\n", 14) ← Disk full ``` ### 7.2 Process inspection ```bash # What files does a process have open? lsof -p 12345 # What ports is a process listening on? ss -tlnp | grep node # Process resource usage top -p 12345 # Or more detailed: cat /proc/12345/status | grep -E "VmRSS|VmSize|Threads" # File descriptors (detect fd leaks) ls /proc/12345/fd | wc -l ``` ### 7.3 tcpdump — Network packet capture ```bash # Capture traffic on port 5432 (PostgreSQL) sudo tcpdump -i any port 5432 -w capture.pcap # Capture HTTP traffic to specific host sudo tcpdump -i any host api.example.com and port 443 # Read captured packets tcpdump -r capture.pcap # Show packet contents as ASCII sudo tcpdump -A -i any port 8080 # Count packets per source IP sudo tcpdump -i any -c 1000 -nn 2>/dev/null | awk '{print $3}' | sort | uniq -c | sort -rn ``` ## 8. Database Debugging ### 8.1 EXPLAIN ANALYZE ```sql -- Always use ANALYZE to get actual execution times EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT) SELECT u.name, count(o.id) as order_count FROM users u LEFT JOIN orders o ON o.user_id = u.id WHERE u.created_at > '2024-01-01' GROUP BY u.id, u.name ORDER BY order_count DESC LIMIT 20; ``` **Reading the output:** ``` Sort (cost=1234..1235 rows=20 width=40) (actual time=45.2..45.3 rows=20 loops=1) Sort Key: (count(o.id)) DESC Sort Method: top-N heapsort Memory: 27kB -> HashAggregate (cost=1200..1220 rows=500 width=40) (actual time=44.1..44.5 rows=500 loops=1) Group Key: u.id -> Hash Right Join (cost=100..800 rows=5000 width=40) (actual time=2.1..30.5 rows=5000 loops=1) -> Seq Scan on orders o (cost=0..500 rows=10000 width=16) (actual time=0.01..10.2 rows=10000 loops=1) ←── PROBLEM: Sequential scan on orders (missing index?) ``` **Key things to look for:** - `Seq Scan` on large tables → missing index - `actual rows` much larger than `rows` estimate → stale statistics (`ANALYZE table`) - `loops=1000` → N+1 query pattern - `Sort Method: external merge Disk` → not enough work_mem ### 8.2 Finding slow queries ```sql -- PostgreSQL: enable slow query log -- In postgresql.conf: -- log_min_duration_statement = 100 (log queries > 100ms) -- Find slow queries with pg_stat_statements SELECT calls, round(total_exec_time::numeric, 2) as total_ms, round(mean_exec_time::numeric, 2) as mean_ms, round(max_exec_time::numeric, 2) as max_ms, left(query, 80) as query FROM pg_stat_statements ORDER BY mean_exec_time DESC LIMIT 20; ``` ### 8.3 Lock debugging ```sql -- Find blocked queries SELECT blocked.pid AS blocked_pid, blocked.query AS blocked_query, blocking.pid AS blocking_pid, blocking.query AS blocking_query, now() - blocked.query_start AS blocked_duration FROM pg_stat_activity blocked JOIN pg_locks bl ON bl.pid = blocked.pid JOIN pg_locks kl ON kl.locktype = bl.locktype AND kl.database IS NOT DISTINCT FROM bl.database AND kl.relation IS NOT DISTINCT FROM bl.relation AND kl.pid != bl.pid JOIN pg_stat_activity blocking ON blocking.pid = kl.pid WHERE NOT bl.granted; ``` ### 8.4 N+1 query detection ```python # Django: use django-debug-toolbar or nplusone # pip install nplusone INSTALLED_APPS = ['nplusone.ext.django', ...] MIDDLEWARE = ['nplusone.ext.django.NPlusOneMiddleware', ...] NPLUSONE_RAISE = True # Raise exception on N+1 # SQLAlchemy: enable echo to see all queries engine = create_engine("postgresql://...", echo=True) # Count queries in tests ``` ## 9. Network Debugging ### 9.1 curl deep dive ```bash # Verbose output — see full request/response headers curl -v https://api.example.com/health # Show timing breakdown curl -w "\ DNS: %{time_namelookup}s\n\ Connect: %{time_connect}s\n\ TLS: %{time_appconnect}s\n\ TTFB: %{time_starttransfer}s\n\ Total: %{time_total}s\n\ HTTP Code: %{http_code}\n\ Size: %{size_download} bytes\n" \ -o /dev/null -s https://api.example.com/health # Test specific HTTP method with headers curl -X POST https://api.example.com/data \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $TOKEN" \ -d '{"key": "value"}' \ -w "\nHTTP %{http_code} in %{time_total}s\n" # Follow redirects curl -L -v https://example.com # Ignore TLS errors (debugging only!) curl -k https://self-signed.example.com # Resolve to specific IP (bypass DNS) curl --resolve api.example.com:443:10.0.0.5 https://api.example.com ``` ### 9.2 DNS debugging ```bash # Query DNS records dig example.com A # IPv4 address dig example.com AAAA # IPv6 address dig example.com CNAME # Canonical name dig example.com MX # Mail servers dig example.com TXT # TXT records (SPF, DKIM, verification) # Use specific DNS server dig @8.8.8.8 example.com # Trace full resolution path dig +trace example.com # Check reverse DNS dig -x 93.184.216.34 # Quick check nslookup example.com host example.com ``` ### 9.3 SSL/TLS debugging ```bash # Check certificate chain openssl s_client -connect example.com:443 -servername example.com < /dev/null # Check certificate expiry openssl s_client -connect example.com:443 2>/dev/null | openssl x509 -noout -dates # Check supported TLS versions openssl s_client -connect example.com:443 -tls1_2 openssl s_client -connect example.com:443 -tls1_3 # Verify certificate chain openssl verify -CAfile chain.pem server.pem ``` ### 9.4 CORS debugging checklist ``` 1. Check the browser console for the exact CORS error message 2. Verify the response includes: - Access-Control-Allow-Origin: <your origin or *> - Access-Control-Allow-Methods: <the method you're using> - Access-Control-Allow-Headers: <any custom headers> 3. For preflight requests (OPTIONS): - Is the server handling OPTIONS requests? - Is it returning 200/204 for OPTIONS? - Is Access-Control-Max-Age set for caching? 4. Common causes: - Origin mismatch (http vs https, port difference, www vs non-www) - Missing Access-Control-Allow-Credentials: true (for cookies) - Wildcard (*) origin not allowed with credentials ``` ## 10. Memory Debugging ### 10.1 Common memory leak patterns | Language | Common cause | Detection | |----------|-------------|-----------| | JavaScript | Event listeners not removed | Heap snapshot comparison | | JavaScript | Closures capturing large objects | Heap snapshot retainer tree | | JavaScript | Detached DOM nodes | DevTools Memory → "Detached" filter | | JavaScript | Growing Map/Set/Array (cache without eviction) | Monitor `process.memoryUsage()` | | Python | Circular references with `__del__` | `gc.get_referrers()`, `objgraph` | | Python | Global/module-level caches | `tracemalloc` | | Go | Goroutine leaks | `runtime.NumGoroutine()`, pprof | | Go | Unclosed channels | `runtime.Stack()` | ### 10.2 JavaScript memory leak debugging workflow ``` 1. Open DevTools → Memory tab 2. Take Heap Snapshot #1 (baseline) 3. Perform the suspected leaking action 5-10 times 4. Force GC (click trash can icon) 5. Take Heap Snapshot #2 6. Select Snapshot #2 → "Objects allocated between Snapshot 1 and Snapshot 2" 7. Sort by "Retained Size" descending 8. Inspect the retainer tree to find what's holding references ``` ### 10.3 Container OOM debugging ```bash # Check if process was OOM killed dmesg | grep -i "oom\|killed" # Check container memory limits docker stats container_name # Check Kubernetes pod events kubectl describe pod my-pod | grep -A5 "Events" # Set memory limits with monitoring docker run --memory=512m --memory-swap=512m my-app # Profile in container docker exec -it container_name node --expose-gc --max-old-space-size=256 app.js ``` ## 11. Performance Profiling ### 11.1 Flame graphs ```bash # Node.js — generate flame graph node --prof app.js # Process the log node --prof-process isolate-*.log > processed.txt # Better: use 0x for automatic flame graph npx 0x app.js # Open the generated HTML flame graph # Python — py-spy flame graph py-spy record -o flamegraph.svg -- python app.py # Open flamegraph.svg in browser # Linux — perf + flame graph perf record -g -p $(pgrep my-app) perf script | stackcollapse-perf.pl | flamegraph.pl > flamegraph.svg ``` **Reading flame graphs:** - X-axis = proportion of time (NOT chronological) - Y-axis = call stack depth (bottom = entry point, top = leaf functions) - Wide bars = functions consuming the most CPU time - Look for "plateaus" — wide, flat tops indicate hot functions ### 11.2 Core Web Vitals debugging | Metric | Target | How to debug | |--------|--------|-------------| | **LCP** (Largest Contentful Paint) | < 2.5s | DevTools → Performance → "LCP" marker; check image loading, font loading, render-blocking resources | | **INP** (Interaction to Next Paint) | < 200ms | DevTools → Performance → click "Interactions"; look for long tasks blocking the main thread | | **CLS** (Cumulative Layout Shift) | < 0.1 | DevTools → Performance → "Layout Shifts"; add explicit width/height to images and ads | ```bash # Measure from command line npx lighthouse https://example.com --output=html --output-path=report.html # Core Web Vitals in JavaScript import { onLCP, onINP, onCLS } from 'web-vitals'; onLCP(console.log); onINP(console.log); onCLS(console.log); ``` ### 11.3 Load testing for debugging ```bash # k6 — modern load testing k6 run --vus 50 --duration 30s script.js # wrk — simple HTTP benchmarking wrk -t4 -c100 -d30s http://localhost:3000/api/users # Apache Bench ab -n 1000 -c 50 http://localhost:3000/api/health ``` ## 12. Logging Strategies ### 12.1 Structured logging ```ts // BAD: unstructured console.log("User " + userId + " failed to login: " + error.message); // GOOD: structured (JSON) import pino from "pino"; const logger = pino({ level: "info" }); logger.error({ event: "login_failed", userId, error: error.message, ip: request.ip, userAgent: request.headers["user-agent"], }, "Login failed for user"); ``` ### 12.2 Log levels | Level | When to use | Example | |-------|-------------|---------| | `fatal` | Application cannot continue | Database connection lost permanently | | `error` | Operation failed, needs attention | Payment processing failed | | `warn` | Unexpected but handled | Rate limit approaching threshold | | `info` | Significant business events | User registered, order placed | | `debug` | Detailed technical info | SQL query executed, cache hit/miss | | `trace` | Very fine-grained | Function entry/exit, variable values | ### 12.3 Correlation IDs ```ts // Generate a unique ID per request for tracing import { randomUUID } from "crypto"; app.use((req, res, next) => { req.requestId = req.headers["x-request-id"]?.toString() ?? randomUUID(); res.setHeader("x-request-id", req.requestId); // Attach to all logs for this request req.log = logger.child({ requestId: req.requestId }); next(); }); // Now all logs from this request are correlated app.get("/api/orders", (req, res) => { req.log.info({ userId: req.user.id }, "Fetching orders"); // ... req.log.info({ count: orders.length }, "Orders fetched"); }); ``` ### 12.4 OpenTelemetry basics ```ts // Distributed tracing across services import { trace, context, SpanStatusCode } from "@opentelemetry/api"; const tracer = trace.getTracer("order-service"); async function processOrder(orderId: string) { return tracer.startActiveSpan("processOrder", async (span) => { try { span.setAttribute("order.id", orderId); const order = await tracer.startActiveSpan("fetchOrder", async (childSpan) => { const result = await db.orders.findById(orderId); childSpan.end(); return result; }); await tracer.startActiveSpan("chargePayment", async (childSpan) => { childSpan.setAttribute("payment.amount", order.total); await paymentService.charge(order); childSpan.end(); }); span.setStatus({ code: SpanStatusCode.OK }); } catch (error) { span.setStatus({ code: SpanStatusCode.ERROR, message: error.message }); span.recordException(error); throw error; } finally { span.end(); } }); } ``` ## 13. Debugging in Production ### 13.1 Debug without redeploying ```ts // Feature flag for debug mode const debugMode = await featureFlags.isEnabled("debug-orders", { userId: currentUser.id, }); if (debugMode) { logger.level = "debug"; logger.debug({ order, paymentResult }, "Order processing debug info"); } ``` ### 13.2 Safe debug endpoints ```ts // Secure debug endpoint (requires admin role + API key) app.get("/debug/connections", requireAdmin, requireApiKey, async (req, res) => { const pool = db.pool; res.json({ total: pool.totalCount, idle: pool.idleCount, waiting: pool.waitingCount, activeQueries: await getActiveQueries(), }); }); ``` ### 13.3 Sentry error tracking ```ts import * as Sentry from "@sentry/node"; Sentry.init({ dsn: process.env.SENTRY_DSN, environment: process.env.NODE_ENV, tracesSampleRate: 0.1, // 10% of transactions beforeSend(event) { // Scrub PII if (event.request?.headers) { delete event.request.headers["authorization"]; delete event.request.headers["cookie"]; } return event; }, }); // Add context to errors Sentry.setUser({ id: user.id, email: user.email }); Sentry.setTag("feature", "checkout"); Sentry.addBreadcrumb({ category: "payment", message: `Charging ${amount} to card ending ${last4}`, level: "info", }); ``` ## 14. Common Pitfalls | Pitfall | Symptom | Investigation Approach | |---------|---------|----------------------| | Debugging the wrong environment | Fix works locally, not in staging | Compare env vars, node versions, OS; use `printenv` diff | | Stale code running | Changes seem to have no effect | Hard refresh (Ctrl+Shift+R); restart dev server; check build output timestamps | | Caching hiding the bug | Bug appears intermittently | Disable all caches (browser, CDN, Redis, ORM query cache); test in incognito | | Race condition | Bug only happens under load or "randomly" | Add logging with timestamps; use `--inspect-brk` to slow execution; test with concurrent requests | | Timezone bug | Dates off by hours; works in some regions | Log `new Date().toISOString()` at each step; check DB timezone settings; use UTC everywhere | | Encoding issue | Garbled text, emoji broken, special chars wrong | Check Content-Type headers; verify UTF-8 at every boundary (DB, API, file I/O) | | Silent error swallowed | Code does nothing; no error visible | Search for empty catch blocks; add `.catch(console.error)` to all promises | | Missing await | Function returns Promise instead of value | TypeScript strict mode; search for `async` functions without `await` on calls | | Circular dependency | Module is undefined at import time | Check import order; use dynamic imports; restructure to break the cycle | | DNS resolution failure | "ENOTFOUND" errors in containers | Check `/etc/resolv.conf`; verify DNS from inside the container with `nslookup` | | Connection pool exhaustion | Timeouts after running fine for hours | Monitor active connections; check for uncommitted transactions; add pool max/idle settings | | Off-by-one error | Wrong count, missing first/last item | Log array lengths and indices; test boundary values: 0, 1, N-1, N | | Environment variable missing | `undefined` used as string, silent failures | Log all env vars on startup (redacted); use zod to validate env at boot | | File descriptor leak | "EMFILE: too many open files" | `lsof -p PID | wc -l`; check for unclosed streams, database connections, or file handles | | Wrong dependency version | Code works in one project but not another | Check `npm ls package-name`; delete `node_modules` and reinstall; check for hoisting issues | | Debugging minified code | Stack traces show line 1, column 43827 | Enable source maps; upload them to Sentry; use `--no-minify` for debugging |

path: scripts/scaffold.sh
content: #!/usr/bin/env bash
# NOTE: On CIFS/SMB mounts, run with: bash scaffold.sh
# Sets up debugging utilities: structured logging, VS Code launch configs, debug helper scripts

set -euo pipefail

PROJECT_DIR="${1:-.}"
echo "Setting up debug utilities in: $PROJECT_DIR"

# Detect project type
LANG="unknown"
if [ -f "$PROJECT_DIR/package.json" ]; then
  LANG="node"
elif [ -f "$PROJECT_DIR/pyproject.toml" ] || [ -f "$PROJECT_DIR/requirements.txt" ] || [ -f "$PROJECT_DIR/setup.py" ]; then
  LANG="python"
fi

echo "Detected project type: $LANG"

# ── Create .vscode directory ──────────────────────────────────────────

mkdir -p "$PROJECT_DIR/.vscode"

# ── VS Code launch.json ──────────────────────────────────────────────

if [ "$LANG" = "node" ]; then
  cat > "$PROJECT_DIR/.vscode/launch.json" << 'LAUNCHEOF'
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug Current File",
      "type": "node",
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal",
      "skipFiles": ["<node_internals>/**", "node_modules/**"]
    },
    {
      "name": "Debug Tests (Jest)",
      "type": "node",
      "request": "launch",
      "program": "${workspaceFolder}/node_modules/.bin/jest",
      "args": ["--runInBand", "--no-coverage", "${relativeFile}"],
      "console": "integratedTerminal",
      "skipFiles": ["<node_internals>/**"]
    },
    {
      "name": "Debug Tests (Vitest)",
      "type": "node",
      "request": "launch",
      "program": "${workspaceFolder}/node_modules/.bin/vitest",
      "args": ["run", "--no-coverage", "${relativeFile}"],
      "console": "integratedTerminal",
      "skipFiles": ["<node_internals>/**"]
    },
    {
      "name": "Attach to Process",
      "type": "node",
      "request": "attach",
      "port": 9229,
      "skipFiles": ["<node_internals>/**", "node_modules/**"]
    }
  ]
}
LAUNCHEOF

elif [ "$LANG" = "python" ]; then
  cat > "$PROJECT_DIR/.vscode/launch.json" << 'LAUNCHEOF'
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug Current File",
      "type": "debugpy",
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal",
      "justMyCode": true
    },
    {
      "name": "Debug FastAPI",
      "type": "debugpy",
      "request": "launch",
      "module": "uvicorn",
      "args": ["app.main:app", "--reload", "--port", "8000"],
      "console": "integratedTerminal",
      "justMyCode": true
    },
    {
      "name": "Debug pytest",
      "type": "debugpy",
      "request": "launch",
      "module": "pytest",
      "args": ["-xvs", "${relativeFile}"],
      "console": "integratedTerminal",
      "justMyCode": true
    },
    {
      "name": "Attach to Remote",
      "type": "debugpy",
      "request": "attach",
      "connect": { "host": "localhost", "port": 5678 },
      "justMyCode": true
    }
  ]
}
LAUNCHEOF

else
  cat > "$PROJECT_DIR/.vscode/launch.json" << 'LAUNCHEOF'
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Debug Current File (Node)",
      "type": "node",
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal",
      "skipFiles": ["<node_internals>/**"]
    },
    {
      "name": "Debug Current File (Python)",
      "type": "debugpy",
      "request": "launch",
      "program": "${file}",
      "console": "integratedTerminal",
      "justMyCode": true
    }
  ]
}
LAUNCHEOF
fi

echo "  Created .vscode/launch.json ($LANG configs)"

# ── VS Code settings.json (debug-friendly) ──────────────────────────

if [ ! -f "$PROJECT_DIR/.vscode/settings.json" ]; then
  cat > "$PROJECT_DIR/.vscode/settings.json" << 'SETEOF'
{
  "debug.console.fontSize": 14,
  "debug.inlineValues": "on",
  "debug.showBreakpointsInOverviewRuler": true,
  "debug.toolBarLocation": "docked"
}
SETEOF
  echo "  Created .vscode/settings.json"
else
  echo "  Skipped .vscode/settings.json (already exists)"
fi

# ── Create scripts directory ─────────────────────────────────────────

mkdir -p "$PROJECT_DIR/scripts"

# ── Debug helper: structured logging setup ───────────────────────────

if [ "$LANG" = "node" ]; then
  cat > "$PROJECT_DIR/scripts/debug-logger.ts" << 'LOGEOF'
/**
 * Structured debug logger with context and levels.
 * Usage:
 *   import { logger } from './scripts/debug-logger';
 *   logger.info('User logged in', { userId: 123 });
 *   logger.error('Failed to fetch', { url, status });
 */

type LogLevel = 'debug' | 'info' | 'warn' | 'error';

const LEVELS: Record<LogLevel, number> = {
  debug: 0,
  info: 1,
  warn: 2,
  error: 3,
};

const currentLevel: LogLevel = (process.env.LOG_LEVEL as LogLevel) || 'info';

function log(level: LogLevel, message: string, context?: Record<string, unknown>): void {
  if (LEVELS[level] < LEVELS[currentLevel]) return;

  const entry = {
    timestamp: new Date().toISOString(),
    level: level.toUpperCase(),
    message,
    ...context,
  };

  const output = JSON.stringify(entry);

  switch (level) {
    case 'error': console.error(output); break;
    case 'warn':  console.warn(output);  break;
    default:      console.log(output);   break;
  }
}

export const logger = {
  debug: (msg: string, ctx?: Record<string, unknown>) => log('debug', msg, ctx),
  info:  (msg: string, ctx?: Record<string, unknown>) => log('info', msg, ctx),
  warn:  (msg: string, ctx?: Record<string, unknown>) => log('warn', msg, ctx),
  error: (msg: string, ctx?: Record<string, unknown>) => log('error', msg, ctx),
};
LOGEOF
  echo "  Created scripts/debug-logger.ts"

elif [ "$LANG" = "python" ]; then
  cat > "$PROJECT_DIR/scripts/debug_logger.py" << 'LOGEOF'
"""
Structured debug logger with context and levels.
Usage:
    from scripts.debug_logger import logger
    logger.info("User logged in", user_id=123)
    logger.error("Failed to fetch", url=url, status=status)
"""

import json
import logging
import sys
from datetime import datetime, timezone


class StructuredFormatter(logging.Formatter):
    def format(self, record: logging.LogRecord) -> str:
        entry = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "level": record.levelname,
            "message": record.getMessage(),
            "logger": record.name,
        }
        # Merge extra context
        if hasattr(record, "context") and isinstance(record.context, dict):
            entry.update(record.context)
        if record.exc_info and record.exc_info[1]:
            entry["exception"] = str(record.exc_info[1])
        return json.dumps(entry)


def get_logger(name: str = "app") -> logging.Logger:
    log = logging.getLogger(name)
    if not log.handlers:
        handler = logging.StreamHandler(sys.stdout)
        handler.setFormatter(StructuredFormatter())
        log.addHandler(handler)
        log.setLevel(logging.DEBUG)
    return log


logger = get_logger()
LOGEOF
  echo "  Created scripts/debug_logger.py"
fi

# ── Debug helper: quick bisect script ────────────────────────────────

cat > "$PROJECT_DIR/scripts/bisect-test.sh" << 'BISEOF'
#!/usr/bin/env bash
# Quick git bisect helper.
# Usage:
#   git bisect start HEAD <last-known-good-commit>
#   git bisect run bash scripts/bisect-test.sh
#
# Edit the TEST_COMMAND below to match your test.

set -euo pipefail

# ── EDIT THIS: the command that should pass ──
TEST_COMMAND="npm test"
# TEST_COMMAND="python -m pytest tests/test_specific.py -x"
# TEST_COMMAND="cargo test specific_test"

echo "=== Bisect test at $(git rev-parse --short HEAD) ==="

if $TEST_COMMAND; then
  echo "PASS — this commit is good"
  exit 0   # good
else
  echo "FAIL — this commit is bad"
  exit 1   # bad
fi
BISEOF
echo "  Created scripts/bisect-test.sh"

# ── Debug helper: timing/profiling wrapper ───────────────────────────

cat > "$PROJECT_DIR/scripts/time-request.sh" << 'TIMEEOF'
#!/usr/bin/env bash
# Time an HTTP request with detailed breakdown.
# Usage: bash scripts/time-request.sh https://localhost:3000/api/health

set -euo pipefail

URL="${1:?Usage: bash scripts/time-request.sh <URL>}"

echo "Timing request to: $URL"
echo ""

curl -o /dev/null -s -w "\
  DNS Lookup:    %{time_namelookup}s\n\
  TCP Connect:   %{time_connect}s\n\
  TLS Handshake: %{time_appconnect}s\n\
  Start Transfer:%{time_starttransfer}s (TTFB)\n\
  Total:         %{time_total}s\n\
  \n\
  HTTP Code:     %{http_code}\n\
  Download Size: %{size_download} bytes\n" "$URL"
TIMEEOF
echo "  Created scripts/time-request.sh"

# ── .env.debug template ─────────────────────────────────────────────

if [ ! -f "$PROJECT_DIR/.env.debug" ]; then
  cat > "$PROJECT_DIR/.env.debug" << 'ENVEOF'
# Debug environment variables — source before debugging
# Usage: source .env.debug && node app.js

# Logging
LOG_LEVEL=debug

# Node.js
NODE_OPTIONS="--enable-source-maps"
# NODE_OPTIONS="--inspect"
# NODE_OPTIONS="--inspect-brk"

# Python
# PYTHONDONTWRITEBYTECODE=1
# PYTHONUNBUFFERED=1
ENVEOF
  echo "  Created .env.debug"
else
  echo "  Skipped .env.debug (already exists)"
fi

# ── Summary ──────────────────────────────────────────────────────────

echo ""
echo "Debug utilities set up successfully!"
echo ""
echo "Files created:"
echo "  .vscode/launch.json    — debugger launch configurations"
echo "  .vscode/settings.json  — debug-friendly editor settings"
if [ "$LANG" = "node" ]; then
  echo "  scripts/debug-logger.ts — structured JSON logger"
elif [ "$LANG" = "python" ]; then
  echo "  scripts/debug_logger.py — structured JSON logger"
fi
echo "  scripts/bisect-test.sh  — git bisect test runner"
echo "  scripts/time-request.sh — HTTP request timing"
echo "  .env.debug              — debug environment variables"
echo ""
echo "Quick start:"
echo "  1. Open the project in VS Code"
echo "  2. Press F5 to start debugging with the launch config"
echo "  3. Set breakpoints by clicking the gutter"


path: references/devtools-cheatsheet.md
content: # Chrome DevTools Cheatsheet

Keyboard shortcuts and panel reference for efficient debugging.

## Global Shortcuts

| Shortcut | Action |
|----------|--------|
| `F12` / `Ctrl+Shift+I` | Open/close DevTools |
| `Ctrl+Shift+J` | Open Console panel |
| `Ctrl+Shift+C` | Inspect element mode |
| `Ctrl+P` | Open file (Sources panel) |
| `Ctrl+Shift+P` | Command palette |
| `Ctrl+F` | Search in current panel |
| `Ctrl+Shift+F` | Search across all sources |
| `Esc` | Toggle Console drawer |

## Elements Panel

| Shortcut | Action |
|----------|--------|
| `Up/Down` | Navigate DOM tree |
| `Left/Right` | Collapse/expand node |
| `Enter` | Edit attribute |
| `F2` | Edit as HTML |
| `Delete` | Remove element |
| `H` | Toggle visibility (display: none) |
| `Ctrl+Z` | Undo DOM change |

### Useful tricks

- **$0** in Console = currently selected element
- **Right-click node → Break on** → subtree modifications / attribute changes
- **Computed tab** → shows final CSS values and which rule applied
- **Event Listeners tab** → shows all event handlers on the element
- **Force state** → right-click → Force state → :hover, :active, :focus

## Console Panel

| Command | Action |
|---------|--------|
| `$0` | Reference to selected element |
| `$_` | Last evaluated result |
| `$$('selector')` | `document.querySelectorAll` shortcut |
| `$x('xpath')` | XPath query |
| `copy(value)` | Copy to clipboard |
| `clear()` | Clear console |
| `dir(object)` | Interactive object tree |
| `table(array)` | Display as table |
| `monitor(fn)` | Log all calls to function |
| `unmonitor(fn)` | Stop monitoring |
| `monitorEvents(el)` | Log all events on element |
| `getEventListeners(el)` | List event listeners |
| `keys(obj)` / `values(obj)` | Object keys/values |
| `queryObjects(Constructor)` | Find all instances |

### Console API

| Method | Use case |
|--------|----------|
| `console.log()` | General output |
| `console.error()` | Error (red, with stack) |
| `console.warn()` | Warning (yellow) |
| `console.table(data, cols)` | Tabular display |
| `console.group('label')` | Start collapsible group |
| `console.groupEnd()` | End group |
| `console.time('label')` | Start timer |
| `console.timeEnd('label')` | End timer, print elapsed |
| `console.assert(cond, msg)` | Log only if condition fails |
| `console.trace()` | Print stack trace |
| `console.count('label')` | Increment and print counter |

## Sources Panel

| Shortcut | Action |
|----------|--------|
| `Ctrl+O` | Open file |
| `Ctrl+G` | Go to line |
| `Ctrl+D` | Select next occurrence |
| `Ctrl+B` | Toggle breakpoint on line |
| `F8` | Resume / pause execution |
| `F10` | Step over |
| `F11` | Step into |
| `Shift+F11` | Step out |
| `Ctrl+'` | Toggle breakpoint enabled/disabled |

### Breakpoint types

| Type | How to set |
|------|-----------|
| Line | Click line number gutter |
| Conditional | Right-click gutter → "Add conditional breakpoint" |
| Logpoint | Right-click gutter → "Add logpoint" (prints to console without pausing) |
| Exception | Click pause icon → "Pause on caught/uncaught exceptions" |
| DOM | Elements panel → right-click element → "Break on" |
| XHR/Fetch | Sources → XHR/fetch Breakpoints → add URL contains |
| Event listener | Sources → Event Listener Breakpoints → expand category |

## Network Panel

| Shortcut | Action |
|----------|--------|
| `Ctrl+E` | Start/stop recording |
| `Ctrl+L` | Clear network log |

### Useful features

- **Filter bar** → type `status-code:500` or `method:POST` or `domain:api.example.com`
- **Throttling** → dropdown to simulate 3G, offline, custom speeds
- **Block request** → right-click → "Block request URL" (test error handling)
- **Copy** → right-click request → "Copy as cURL" / "Copy as fetch"
- **Override response** → right-click → "Override content" (mock APIs locally)

### Timing breakdown

```
DNS Lookup ──→ TCP Connect ──→ TLS Handshake ──→ Request Sent ──→ Waiting (TTFB) ──→ Content Download
```

## Performance Panel

| Shortcut | Action |
|----------|--------|
| `Ctrl+E` | Start/stop recording |
| `Ctrl+Shift+E` | Record + reload page |

### What to look for

| Issue | Where to check |
|-------|---------------|
| Long tasks (>50ms) | Main thread → red triangles |
| Layout thrashing | "Layout" events in flame chart |
| Forced synchronous layout | Purple bars with warning icon |
| Expensive paint | Green bars in "Frames" row |
| JavaScript bottleneck | Wide yellow bars in flame chart → zoom in |
| Idle time | Gray gaps between tasks |

## Memory Panel

### Heap Snapshot workflow

1. Take Snapshot 1 (baseline)
2. Do the suspected leaking action
3. Force GC (trash can icon)
4. Take Snapshot 2
5. Select Snapshot 2 → view: "Comparison" with Snapshot 1
6. Sort by "# Delta" or "Size Delta"

### What to look for

| Problem | How to identify |
|---------|----------------|
| Detached DOM nodes | Filter "Detached" in heap snapshot |
| Growing arrays | Compare snapshots → Array size increases |
| Event listener leaks | `getEventListeners($0)` shows unexpected listeners |
| Closure leaks | Retainer tree shows closure holding large objects |

## Application Panel

| Feature | What it shows |
|---------|--------------|
| Local Storage | Key-value pairs for the domain |
| Session Storage | Session-scoped key-value pairs |
| IndexedDB | Structured database contents |
| Cookies | All cookies with details |
| Service Workers | Registered workers, push subscriptions |
| Cache Storage | Cached resources from Service Worker |


path: references/profiling-tools-guide.md
content: # Profiling Tools Guide

Comparison of profiling tools across languages and runtimes for CPU, memory, and I/O analysis.

---

## Python Profiling

### py-spy (Sampling Profiler)

**Install:** `pip install py-spy`

| Feature | Details |
|---------|---------|
| Type | Sampling profiler (low overhead) |
| Attach to running | Yes — `py-spy top --pid PID` |
| Flame graph | `py-spy record -o flame.svg -- python app.py` |
| Native extensions | Shows C/C++ frames with `--native` |
| Subprocesses | `--subprocesses` flag |
| Overhead | ~2-5% |

```bash
# Live top-like view of running process
py-spy top --pid 12345

# Record flame graph for 30 seconds
py-spy record -o profile.svg --duration 30 --pid 12345

# Profile a command from start
py-spy record -o flame.svg -- python manage.py runserver

# Show only lines taking >5% of time
py-spy top --pid 12345 --rate 100
```

### cProfile / profile (Built-in)

```bash
# Run with cProfile
python -m cProfile -s cumulative app.py

# Save to file and analyse with pstats
python -m cProfile -o output.prof app.py
python -c "
import pstats
p = pstats.Stats('output.prof')
p.sort_stats('cumulative').print_stats(20)
"

# Visualise with snakeviz
pip install snakeviz
snakeviz output.prof   # opens browser
```

| Option | Description |
|--------|-------------|
| `-s cumulative` | Sort by cumulative time |
| `-s tottime` | Sort by total time in function |
| `-s calls` | Sort by call count |
| `-o file.prof` | Save binary profile output |

### Scalene (CPU + Memory + GPU)

**Install:** `pip install scalene`

```bash
# Profile with CPU, memory, and GPU breakdown
scalene app.py

# Focus on specific file
scalene --cpu --memory --reduced-profile app.py

# Profile as JSON
scalene --json --outfile profile.json app.py
```

| Feature | Details |
|---------|---------|
| CPU time | Separates Python vs native (C) time |
| Memory | Tracks allocations per line |
| GPU | CUDA memory tracking |
| Copy volume | Identifies excessive copying |
| Overhead | ~20-30% |

### memray (Memory Profiler)

**Install:** `pip install memray`

```bash
# Record memory allocations
memray run app.py

# Generate flame graph
memray flamegraph output.bin -o memory.html

# Live view while running
memray run --live app.py

# Show top allocations
memray summary output.bin

# Generate table of allocations by location
memray table output.bin
```

---

## Node.js Profiling

### node --prof (V8 Built-in)

```bash
# Record V8 profile (creates isolate-*.log)
node --prof app.js

# Process the log into readable output
node --prof-process isolate-0x*.log > processed.txt

# Key sections in output:
#   [Summary]        — time breakdown by category
#   [JavaScript]     — JS function hotspots
#   [C++]            — native code hotspots
#   [Bottom up]      — callee → caller chains
```

### node --inspect (Chrome DevTools)

```bash
# Start with inspector
node --inspect app.js

# Break on first line
node --inspect-brk app.js

# Then open: chrome://inspect in Chrome
# Click "inspect" on the listed target
```

**DevTools profiling workflow:**
1. Open chrome://inspect → click "inspect"
2. Go to **Performance** tab → click Record
3. Do the action you want to profile
4. Stop recording → analyse flame chart

### 0x (Flame Graph Generator)

**Install:** `npm install -g 0x`

```bash
# Generate flame graph (opens browser automatically)
0x app.js

# With custom port
0x -p 3000 app.js

# Output as SVG instead of interactive HTML
0x --output-dir ./profiles app.js

# Profile for specific duration
0x --collect-delay 5000 app.js
```

| Feature | Details |
|---------|---------|
| Type | Sampling via V8 profiler |
| Output | Interactive HTML flame graph |
| Overhead | Low when sampling |
| Best for | Finding CPU hotspots in Node.js |

### clinic.js (Diagnostic Suite)

**Install:** `npm install -g clinic`

```bash
# Doctor — auto-detects common issues
clinic doctor -- node app.js

# Flame — CPU flame graph
clinic flame -- node app.js

# Bubbleprof — async I/O analysis
clinic bubbleprof -- node app.js

# Heap profiler — memory analysis
clinic heapprofiler -- node app.js
```

| Tool | Detects |
|------|---------|
| `doctor` | Event loop delays, I/O issues, GC pressure, active handles |
| `flame` | CPU hotspots (built on 0x) |
| `bubbleprof` | Async flow bottlenecks, I/O patterns |
| `heapprofiler` | Memory leaks, allocation hotspots |

---

## System-Level Profiling (Linux)

### perf (Linux Performance Counter)

```bash
# Record CPU profile of a process
sudo perf record -g -p PID

# Record a specific command
sudo perf record -g -- node app.js

# Show report (interactive TUI)
sudo perf report

# Show top functions
sudo perf top -p PID

# Generate flame graph (with Brendan Gregg's tools)
sudo perf record -F 99 -g -p PID -- sleep 30
sudo perf script > out.perf
./stackcollapse-perf.pl out.perf > out.folded
./flamegraph.pl out.folded > flame.svg

# Stat summary (counters: cache misses, branch mispredictions, etc.)
sudo perf stat -d -- ./myprogram
```

| Feature | Details |
|---------|---------|
| Type | Hardware performance counters + software events |
| Overhead | Very low (~1-3%) |
| Scope | System-wide or per-process |
| Output | Interactive TUI, scriptable text |
| Root required | Yes (or `perf_event_paranoid` sysctl) |

### Flame Graphs (Brendan Gregg Method)

```bash
# Clone the flame graph tools
git clone https://github.com/brendangregg/FlameGraph.git

# Workflow: perf → collapse → flamegraph
sudo perf record -F 99 -a -g -- sleep 30
sudo perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > flame.svg

# For Node.js (using perf with --perf-basic-prof)
node --perf-basic-prof app.js &
sudo perf record -F 99 -p $! -g -- sleep 30
sudo perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > node-flame.svg
```

**Reading a flame graph:**

| Element | Meaning |
|---------|---------|
| X axis | Alphabetical stack sort (NOT time) |
| Y axis | Stack depth (bottom = entry point) |
| Width | % of total samples (wider = more CPU time) |
| Colour | Random warm palette (no special meaning) |
| Plateau | Wide flat top = function spending lots of CPU |
| Narrow tower | Deep call stack but fast |

### strace / ltrace

```bash
# Trace system calls
strace -p PID
strace -e trace=network -p PID      # network calls only
strace -e trace=file -p PID         # file operations only
strace -c -p PID                    # summary: call counts and times
strace -T -p PID                    # show time spent in each syscall
strace -f -p PID                    # follow child processes/threads

# Trace library calls
ltrace -p PID
ltrace -c -p PID                    # summary
```

---

## Database Query Profiling

### PostgreSQL

```sql
-- Enable timing
\timing on

-- Basic EXPLAIN
EXPLAIN SELECT * FROM users WHERE email = '[REDACTED_SECRET]';

-- EXPLAIN with actual execution
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT * FROM users WHERE email = '[REDACTED_SECRET]';

-- Log slow queries (in postgresql.conf)
-- log_min_duration_statement = 100   -- log queries > 100ms

-- pg_stat_statements extension (top queries by time)
SELECT query, calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;
```

### MySQL

```sql
-- Enable profiling for session
SET profiling = 1;

-- Run your query
SELECT * FROM users WHERE email = '[REDACTED_SECRET]';

-- Show profile
SHOW PROFILES;
SHOW PROFILE FOR QUERY 1;

-- EXPLAIN
EXPLAIN ANALYZE SELECT * FROM users WHERE email = '[REDACTED_SECRET]';

-- Slow query log (in my.cnf)
-- slow_query_log = 1
-- long_query_time = 0.1
```

---

## Tool Selection Matrix

| Scenario | Python | Node.js | System |
|----------|--------|---------|--------|
| **CPU hotspot** | py-spy | 0x / clinic flame | perf |
| **Memory leak** | memray / Scalene | clinic heapprofiler / --inspect | Valgrind |
| **Async bottleneck** | — | clinic bubbleprof | strace |
| **Line-by-line** | Scalene / cProfile | --prof-process | perf annotate |
| **Production safe** | py-spy (low overhead) | 0x (sampling) | perf record |
| **Quick overview** | cProfile | clinic doctor | perf stat |
| **Flame graph** | py-spy record | 0x / clinic flame | perf + FlameGraph |
| **GPU profiling** | Scalene | — | nvidia-smi / nsys |

## Quick Decision Tree

```
Is the app slow?
├── CPU-bound? ──→ Use flame graph (py-spy / 0x / perf)
├── Memory issue? ──→ Use heap profiler (memray / clinic heapprofiler)
├── I/O-bound? ──→ Use async profiler (clinic bubbleprof / strace)
├── Database slow? ──→ EXPLAIN ANALYZE + pg_stat_statements
└── Network slow? ──→ DevTools Network tab + curl timing
```

## Common Pitfalls

| Pitfall | Solution |
|---------|----------|
| Profiling in development only | Production has different behaviour — use low-overhead tools (py-spy, perf) in prod |
| Ignoring warm-up | First requests include JIT compilation — run warm-up before profiling |
| Micro-benchmarking | Isolated benchmarks miss system interactions — profile realistic workloads |
| Not using flame graphs | Flat profiles hide call-chain context — always generate flame graphs for CPU issues |
| Profiling with debugger attached | Debuggers add overhead — profile without breakpoints or debug mode |
| Root cause vs symptom | High CPU in GC means memory issue, not CPU issue — look deeper |
| Forgetting to check I/O | CPU profilers miss I/O waits — combine with strace or async profilers |


path: examples/debug-checklist.md
content: # Debugging Workflow Checklist

Step-by-step checklist for systematic bug investigation.

## Phase 1: Reproduce

- [ ] Can you trigger the bug reliably? (Y/N)
- [ ] Minimal steps to reproduce documented?
- [ ] Environment noted? (OS, runtime version, browser, config)
- [ ] Is it deterministic or intermittent?
- [ ] If intermittent: what's the pattern? (load, timing, specific data)

## Phase 2: Gather Information

- [ ] Read the **full** error message and stack trace
- [ ] Check application logs (last 30 minutes)
- [ ] Check monitoring dashboards for anomalies
- [ ] Check recent deployments/changes (`git log --oneline -20`)
- [ ] Check recent config changes
- [ ] Ask: "What changed recently?"

## Phase 3: Isolate

- [ ] Which component is affected? (frontend, backend, database, network, infra)
- [ ] Does it happen in all environments? (local, staging, production)
- [ ] Does it happen for all users or specific ones?
- [ ] Does it happen with all data or specific inputs?
- [ ] Binary search: can you narrow it to a specific module/function?

## Phase 4: Hypothesize

Write down at least 3 possible causes:

1. [ ] Hypothesis: ________________________________
2. [ ] Hypothesis: ________________________________
3. [ ] Hypothesis: ________________________________

For each hypothesis, define a test:
- "If it's [hypothesis], then [this test] should show [this result]"

## Phase 5: Test and Fix

- [ ] Test each hypothesis (30 min timebox per hypothesis)
- [ ] Root cause identified?
- [ ] Fix is minimal and targeted? (don't fix "while you're in there")
- [ ] Fix doesn't introduce new issues? (run full test suite)

## Phase 6: Verify and Prevent

- [ ] Original reproduction steps no longer trigger the bug
- [ ] No regressions in related functionality
- [ ] Added a test that catches this specific bug
- [ ] Updated documentation/runbook if applicable
- [ ] Shared findings with the team (Slack, PR description, post-mortem)

## Quick Reference: Common Bug Categories

| Category | Symptoms | First check |
|----------|----------|-------------|
| **Logic error** | Wrong result, no error | Add logging at decision points |
| **Null/undefined** | TypeError, "cannot read property" | Check data flow, API responses |
| **Race condition** | Intermittent, timing-dependent | Add locks/queuing, check async code |
| **Configuration** | Works locally, fails in staging/prod | Diff environment variables and configs |
| **Dependency** | Broke after update | Check `git diff` on lock files |
| **Memory leak** | Slow degradation over time | Monitor RSS, take heap snapshots |
| **N+1 query** | Slow page load, high DB load | Enable query logging, count queries |
| **Encoding** | Garbled text, broken characters | Check UTF-8 at every boundary |
| **Timezone** | Dates off by hours | Use UTC everywhere, check DB timezone |
| **Permission** | "Access denied", 403 | Check file permissions, IAM roles, CORS |
| **DNS/Network** | "ENOTFOUND", timeouts | `curl -v`, `dig`, check firewall |
| **Disk space** | "ENOSPC", write failures | `df -h`, check log rotation |

## Rubber Duck Protocol

When stuck for more than 30 minutes:

1. Open a blank document
2. Write: "The code should do: ___"
3. Write: "Instead it does: ___"
4. Write: "I've checked: ___"
5. Write: "I haven't checked: ___" ← **Start here**


path: examples/git-bisect-example.sh
content: #!/usr/bin/env bash
# NOTE: On CIFS/SMB mounts, run with: bash git-bisect-example.sh
# Demonstrates automated git bisect to find the commit that introduced a bug
#
# Usage:
#   bash git-bisect-example.sh <good-ref> <bad-ref> <test-command>
#
# Examples:
#   bash git-bisect-example.sh v1.0.0 HEAD "npm test"
#   bash git-bisect-example.sh abc123 HEAD "node -e 'require(\"./dist\").validate(\"test\")'"
#   bash git-bisect-example.sh v2.0 main "python -m pytest tests/test_auth.py -x"

set -euo pipefail

GOOD_REF="${1:-}"
BAD_REF="${2:-HEAD}"
TEST_CMD="${3:-}"

if [ -z "$GOOD_REF" ] || [ -z "$TEST_CMD" ]; then
  echo "Usage: bash git-bisect-example.sh <good-ref> <bad-ref> <test-command>"
  echo ""
  echo "Arguments:"
  echo "  good-ref     A commit/tag where the code works correctly"
  echo "  bad-ref      A commit/tag where the bug exists (default: HEAD)"
  echo "  test-command  Command that exits 0 if OK, non-zero if bug present"
  echo ""
  echo "Examples:"
  echo "  bash git-bisect-example.sh v1.0.0 HEAD 'npm test'"
  echo "  bash git-bisect-example.sh abc123 main 'python -m pytest tests/ -x'"
  echo ""
  exit 1
fi

echo "========================================="
echo "  Git Bisect — Automated Bug Finder"
echo "========================================="
echo ""
echo "Good commit:  $GOOD_REF"
echo "Bad commit:   $BAD_REF"
echo "Test command: $TEST_CMD"
echo ""

# Ensure we're in a git repo
if ! git rev-parse --is-inside-work-tree &>/dev/null; then
  echo "ERROR: Not inside a git repository."
  exit 1
fi

# Check for uncommitted changes
if ! git diff --quiet || ! git diff --cached --quiet; then
  echo "WARNING: You have uncommitted changes."
  echo "Stashing changes before bisect..."
  git stash push -m "bisect-auto-stash"
  STASHED=true
else
  STASHED=false
fi

# Start bisect
echo ""
echo "Starting bisect..."
git bisect start "$BAD_REF" "$GOOD_REF"

# Create a temporary test script that handles build + test
BISECT_SCRIPT=$(mktemp /tmp/bisect-test-XXXXXX.sh)
cat > "$BISECT_SCRIPT" << SCRIPTEOF
#!/usr/bin/env bash
set -e

# If there's a build step, try it (but skip commit if build fails)
if [ -f "package.json" ] && grep -q '"build"' package.json 2>/dev/null; then
  npm run build 2>/dev/null || exit 125  # 125 = skip this commit
fi

if [ -f "Makefile" ] && grep -q '^build:' Makefile 2>/dev/null; then
  make build 2>/dev/null || exit 125
fi

# Run the actual test
$TEST_CMD
SCRIPTEOF
chmod +x "$BISECT_SCRIPT" 2>/dev/null || true

echo "Running automated bisect..."
echo ""

# Run bisect with the test script
git bisect run bash "$BISECT_SCRIPT"

# Capture the result
RESULT=$?

echo ""
echo "========================================="
if [ $RESULT -eq 0 ]; then
  echo "  Bisect complete!"
  echo ""
  echo "  First bad commit:"
  git bisect log | tail -1
  echo ""
  echo "  Commit details:"
  git log --oneline -1
else
  echo "  Bisect encountered an error (exit code: $RESULT)"
fi
echo "========================================="

# Clean up
rm -f "$BISECT_SCRIPT"
git bisect reset

# Restore stashed changes
if [ "$STASHED" = true ]; then
  echo ""
  echo "Restoring stashed changes..."
  git stash pop
fi

echo ""
echo "Done. Working tree restored to original state."


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: clean | SkillSpector: suspicious (HIGH)