Detecting webshells for fun and profit
By: Uri Fleyder-Kotler
Tags: Web Server, Webshell, Linux, Incident Response, Cybersecurity

Analyze the Web Server Logs

As the amount of data in web server logs may be overwhelming sometimes, especially in high traffic servers, my preferred tactic is analyzing them using the bottom-up approach.

Let’s start with only one log file and check only one date in each iteration, focusing only on one HTTP method that is quite often utilized by Cybercriminals to achieve their goals using your server:

grep “02/May” /var/www/vhosts/*/logs/access_ssl_log.processed | grep POST

A similar technique may be used to investigate additional HTTP methods, or simply all the methods excluding the most common method — GET:

grep “02/May” /var/www/vhosts/*/logs/access_ssl_log.processed | grep -v GET

You will receive a long list in the following format:

<IP Address> — — <Date and Time> “<HTTP Method> /<Requested Resource> <HTTP/Protocol Version>” <Status Code> <Response Size> “<Referer>” “User-Agent”

I will focus on the following data points:

IP Address

Requested Resource

Status Code

User-Agent

Let’s review the unique requests focusing only on the combination of IP Address, Requested Resource, Status Code, and User-Agent:

grep “02/May” /var/www/vhosts/*/logs/access_ssl_log.processed | grep POST | cut -d ‘ ‘ -f 1,7,9,12 | sort -u

For popular CMS platforms, such as WordPress, we can filter out some of the common file names and plugin generated requests to look for the outliers:

grep “02/May” /var/www/vhosts/*/logs/access_ssl_log.processed | grep POST | cut -d ‘ ‘ -f 1,7,9,12 | grep -E -v “wp-cron.php|admin-ajax.php|xmlrpc.php|wp-login.php|admin-post.php|wordfence_syncAttackData” | sort -u

And another useful combination of “locate” + “xargs” + “grep” to locate all the logs and execute our search pattern on each one of the discovered log files:

locate “/access.” | xargs grep “02/May” | grep -v GET | cut -d ‘ ‘ -f 1,7,9,12 | grep -E -v “wp-cron.php|admin-ajax.php|xmlrpc.php|wp-login.php|admin-post.php|wordfence_syncAttackData” | sort -u locate “/access_” | xargs grep “02/May” | grep -v GET | cut -d ‘ ‘ -f 1,7,9,12 | grep -E -v “wp-cron.php|admin-ajax.php|xmlrpc.php|wp-login.php|admin-post.php|wordfence_syncAttackData” | sort -u

At this point, I usually start to see very suspicious-looking requests, exposing the paths of the webshells that Cybercriminals planted on the server.

For persistency purposes, oftentimes, once a Cybercriminal found and exploited a vulnerability against an unpatched component of a given website (e.g., outdated WP version, theme, or plugin) he will plant many webshells in different directories that he may access using the same set of permissions you have granted to your WP installation — this is why following the principle of least privilege is so important when installing software.

Once you detect and remove some of the malicious files, you will notice how the cybercriminal tries to reach his removed webshells multiple times, before revealing the path of additional files he has planted on your server.

For example, in one of my recent engagements, the technique described above helped me to find out that a single IP address (“35.198.236.40”) executed file enumeration attack using 570 unique file names across dozens of domains on the same hosting server, many of which I identified as common webshell names based on my previous engagements.

The following pattern includes another filter to find only the requests that the server returned successfully (Status Code = 200).

It can assist you to find out which webshells are active at this point and require you to remove them from the server:

locate “/access.” | xargs grep “02/May” | grep -v GET | cut -d ‘ ‘ -f 1,7,9,12 | grep -E -v “wp-cron.php|admin-ajax.php|xmlrpc.php|wp-login.php|admin-post.php|wordfence_syncAttackData” | grep “ 200 “ | sort -u locate “/access_” | xargs grep “02/May” | grep -v GET | cut -d ‘ ‘ -f 1,7,9,12 | grep -E -v “wp-cron.php|admin-ajax.php|xmlrpc.php|wp-login.php|admin-post.php|wordfence_syncAttackData” | grep “ 200 “ | sort -u

Overall, I don’t believe in blocking “bad” IP addresses, at least not as long as they’re not involved in a denial of service attack against your server.

When you discover any information about your attackers, like a list of IP addresses and unique User-Agents they’re using, you should leverage this information against them, instead of simply blocking them using your WAF and losing your advantage.

Know about them, but don’t let them know that you know about them, this is the best way to protect your assets in the digital world, IMO.