Skill: kiaraho/sys-guard-linux-remediator (version 1.0.0)

--- name: sys-guard-linux-remediator description: Host-based Linux incident response and remediation skill focused on precise threat detection, forensic-safe data collection, firewall control (iptables/nftables), integrity validation, and controlled remediation while preserving system stability. metadata: author: Edwin Kairu ([REDACTED_SECRET]) --- # Linux Threat Mitigation and Incident Remediation (Hardened Edition) This skill provides a structured, forensically-aware framework for analyzing and securing a Linux host during or after a security event. It emphasizes: - Non-destructive evidence collection - Accurate threat detection - Firewall-aware containment - Integrity verification - Controlled, reversible remediation - Distribution-aware command usage --- # Environment Context ## Supported Systems - Debian / Ubuntu - RHEL / CentOS / Rocky / Alma - Fedora - Arch Linux (limited package guidance) ## Execution Assumptions - Shell: `bash` or POSIX `sh` - Privilege: Root or sudo - Host-level access (NOT container-restricted environments) - systemd-based systems preferred > ⚠️ If running inside Docker, Kubernetes, LXC, or other containers, firewall, audit, and service commands may not reflect the host system. --- # Firewall Architecture Awareness Modern Linux systems may use: - `iptables-legacy` - `iptables-nft` (compatibility wrapper) - Native `nftables` - `firewalld` (RHEL-family default) ## Identify Firewall Backend ```bash iptables --version which nft systemctl status firewalld ``` If nftables is active: ```bash nft list ruleset ``` Do NOT assume `iptables -L` represents the full firewall state. --- # Logging Differences by Distribution | Distribution | Primary Log File | |--------------|------------------| | Ubuntu/Debian | `/var/log/syslog` | | RHEL/CentOS/Fedora | `/var/log/messages` | | All modern systemd | `journalctl` | Always prefer: ```bash journalctl -xe ``` --- # Operational Toolkit (Hardened) ## 1. Network Inspection ### Listening Services ```bash ss -tulpn ``` ### Active Connections ```bash ss -antp | grep ESTABLISHED ``` ### Firewall State #### iptables ```bash iptables -L -n -v --line-numbers iptables -S ``` #### nftables ```bash nft list ruleset ``` ### Local Service Enumeration (Low Noise) ```bash ss -lntup ``` Avoid unnecessary full scans of localhost unless required. ### Conservative Network Scan ```bash nmap -sV -T3 -p- localhost ``` ### Packet Capture (Short Snapshot) ```bash tcpdump -i any -nn -c 100 ``` --- ## 2. Process & Runtime Analysis ### Process Tree ```bash ps auxww --forest ``` ### High CPU / Memory ```bash top ``` ### Open File Handles ```bash lsof -p <PID> ``` ### System Call Trace (Caution: Alters Timing) ```bash strace -p <PID> ``` > ⚠️ `strace` may change process behavior. Use carefully during live compromise. ### Kernel Modules ```bash lsmod ``` ### Kernel Messages ```bash dmesg | tail -50 ``` --- ## 3. Rootkit & Malware Scanning ### Rootkit Scanners ```bash rkhunter --check chkrootkit ``` > May produce false positives. Validate findings manually. ### Antivirus Scan (Targeted) ```bash clamscan -r /home ``` Use selectively; large scans increase I/O and may alter access timestamps. ### Lynis System Audit ```bash lynis audit system ``` --- ## 4. File Integrity & Package Verification ### AIDE (After Initialization) Install: ```bash apt install aide # or dnf install aide ``` Initialize: ```bash aideinit mv /var/lib/aide/aide.db.new.gz /var/lib/aide/aide.db.gz ``` Run Check: ```bash aide --check ``` ### RHEL Package Verification ```bash rpm -Va ``` ### Debian Package Verification ```bash apt install debsums debsums -s ``` --- ## 5. Forensic Analysis (Didier Stevens Suite) Install: ```bash sudo mkdir -p /opt/forensics sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/base64dump.py sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/re-search.py sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/zipdump.py sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/1768.py sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/pdf-parser.py sudo wget -P /opt/forensics https://raw.githubusercontent.com/DidierStevens/DidierStevensSuite/master/oledump.py sudo chmod +x /opt/forensics/*.py ``` ### Decode Base64 ```bash python3 /opt/forensics/base64dump.py file.txt ``` ### IOC Search ```bash python3 /opt/forensics/re-search.py -n ipv4 logfile ``` ### Inspect ZIP (No Extraction) ```bash python3 /opt/forensics/zipdump.py suspicious.zip ``` ### Extract Cobalt Strike Beacon Config ```bash python3 /opt/forensics/1768.py payload.bin ``` ### Inspect Office/PDF Documents ```bash python3 /opt/forensics/pdf-parser.py file.pdf python3 /opt/forensics/oledump.py file.doc ``` > Static inspection only. Never execute suspicious files. --- ## 6. Authentication & User Activity ### Current Sessions ```bash who -a ``` ### Login History ```bash last -a ``` ### Failed SSH Logins Ubuntu/Debian: ```bash journalctl -u ssh.service | grep "Failed password" ``` RHEL/Fedora: ```bash journalctl -u sshd.service | grep "Failed password" ``` ### Sudo Activity ```bash journalctl _COMM=sudo ``` ### Audit Logs ```bash ausearch -m USER_AUTH,USER_LOGIN,USER_CHAUTHTOK ``` --- # Controlled Remediation ## Blocking an IP ### iptables (Immediate) ```bash iptables -I INPUT 1 -s <IP> -j DROP ``` ### nftables ```bash nft add rule inet filter input ip saddr <IP> drop ``` If firewalld is active: ```bash firewall-cmd --add-rich-rule='rule family="ipv4" source address="<IP>" drop' ``` --- ## Persisting Firewall Rules iptables (Debian): ```bash netfilter-persistent save ``` iptables (manual save): ```bash iptables-save > /etc/iptables/rules.v4 ``` firewalld: ```bash firewall-cmd --runtime-to-permanent ``` nftables: ```bash nft list ruleset > /etc/nftables.conf ``` --- ## Process Containment Strategy Preferred escalation: 1. Observe 2. `kill -TERM <PID>` 3. If required: `kill -STOP <PID>` for analysis 4. Use `kill -KILL <PID>` only if necessary Avoid `killall` or broad `pkill`. --- ## Service Isolation ```bash systemctl stop <service> systemctl disable <service> systemctl mask <service> ``` --- # Persistence & Backdoor Checks ### Cron Jobs ```bash crontab -l ls -lah /etc/cron* ``` ### Systemd Persistence ```bash ls -lah /etc/systemd/system/ ``` ### Startup Scripts ```bash cat /etc/rc.local ``` --- # SELinux Awareness (RHEL/Fedora) Check status: ```bash getenforce ``` Review denials: ```bash ausearch -m AVC ``` --- # Forensic Hygiene 1. Never execute suspicious binaries. 2. Preserve evidence before deletion: ```bash sha256sum file mkdir -p /root/quarantine mv file /root/quarantine/file.vir ``` 3. Log every remediation step: ```bash date -u ``` Document: - Timestamp - Command executed - Observed outcome --- # Usage Examples ## Routine Audit - Run `lynis audit system` - Verify no unknown listening services - Check for modified system binaries ## Active Threat - Identify high CPU process - Capture short `tcpdump` - Extract file hash - Contain IP via firewall - Preserve malicious artifact ## Suspicious File - Use `zipdump` - Extract hash - Move to quarantine - Search logs for execution attempts --- # Safety Guardrails These guardrails are mandatory and apply to all remediation activity. Their purpose is to prevent self-inflicted outages, preserve forensic integrity, and ensure reversible, controlled incident response. --- ## 1. State Verification (Pre- and Post-Change Validation) Before executing any remediation command: 1. Record timestamp (UTC): ```bash date -u ``` 2. Run a discovery command to capture current state: - Network: `ss -tulpn` - Active connections: `ss -antp` - Firewall (iptables): `iptables -L -n -v` - Firewall (nftables): `nft list ruleset` - firewalld: `firewall-cmd --list-all` After remediation: 3. Re-run the same discovery command. 4. Compare state change and confirm: - Intended effect achieved - No unintended service disruption - No management lockout (e.g., SSH access intact) Never assume a command succeeded without verifying its effect. --- ## 2. No Wildcards or Broad Termination To prevent catastrophic system damage: - NEVER use: - `rm -rf *` - `rm -rf /` - `killall` - Broad `pkill` patterns - Unbounded globbing in sensitive directories - Always: - Use absolute file paths (e.g., `/tmp/malware.bin`) - Target explicit PIDs (`kill -TERM <PID>`) - Confirm file existence with `ls -lah <file>` - Hash suspicious files before modification: ```bash sha256sum <file> ``` Wildcard deletions and pattern-based termination are prohibited during incident response. --- ## 3. Persistence & Re-Spawn Inspection After containment of a malicious process or service, immediately inspect for persistence mechanisms. ### Check: #### Cron Jobs ```bash crontab -l ls -lah /etc/cron* ``` #### systemd Services & Timers ```bash systemctl list-unit-files --type=service systemctl list-timers --all ls -lah /etc/systemd/system/ ``` #### Init Scripts ```bash ls -lah /etc/init.d/ cat /etc/rc.local ``` #### User-Level Persistence ```bash ls -lah ~/.config/systemd/user/ ``` #### SSH Backdoors ```bash cat ~/.ssh/authorized_keys ``` After removal of malicious artifacts: - Run integrity verification: ```bash aide --check ``` - On RHEL-based systems: ```bash rpm -Va ``` - On Debian-based systems: ```bash debsums -s ``` Do not consider a threat eradicated until persistence mechanisms are eliminated. --- ## 4. Firewall Rule Safety & Persistence ### A. Anti-Lockout Requirement Before modifying firewall rules: 1. Confirm SSH listening port: ```bash ss -tulpn | grep ssh ``` 2. Confirm an explicit ACCEPT rule exists for: - Current management IP - SSH port NEVER: ```bash iptables -F ``` NEVER set a default DROP policy without verifying SSH access rule exists. --- ### B. Immediate vs Persistent Rules Firewall rule changes are runtime by default and may not survive reboot. #### iptables (Debian/Ubuntu) Runtime only until saved: ```bash iptables-save > /etc/iptables/rules.v4 ``` If using netfilter-persistent: ```bash netfilter-persistent save ``` #### RHEL (legacy iptables service) ```bash service iptables save ``` #### firewalld Runtime-to-permanent: ```bash firewall-cmd --runtime-to-permanent ``` #### nftables Persist ruleset: ```bash nft list ruleset > /etc/nftables.conf ``` Document: - Whether rule is temporary or permanent - Location of saved configuration - Verification after reboot (if applicable) --- ## 5. Forensic Preservation Before Destruction Before deleting or killing: 1. Hash the artifact: ```bash sha256sum <file> ``` 2. Move to quarantine: ```bash mkdir -p /root/quarantine mv <file> /root/quarantine/<file>.vir ``` 3. Record: - Timestamp (UTC) - Original path - Hash value - Reason for containment Avoid `kill -9` unless absolutely required. Prefer: 1. `kill -TERM <PID>` 2. `kill -STOP <PID>` (if forensic inspection needed) 3. `kill -KILL <PID>` only as last resort --- ## 6. Change Logging Requirement Every remediation action must include: - `date -u` - Command executed - Justification - Observed outcome - Updated risk level (if applicable) Remediation without documentation is non-compliant. --- ## 7. Minimal-Impact Principle All actions must follow: - Smallest necessary change - Reversible where possible - No broad configuration resets - No service restarts without justification - No system-wide scans during active compromise unless scoped Contain first. Eradicate methodically. Recover cautiously.


Security scan verdicts — ClawScan: clean | static analysis: clean | VirusTotal: suspicious | SkillSpector: clean (LOW)