Hack Like a Pro: An Introduction to Regular Expressions (Regex) « Null Byte :: WonderHowTo

Welcome back, my novice hackers!This next subject might seem a bit obscure, but I promise you, this lesson will benefit you significantly either as a hacker or system admin. This tutorial will cover what is usually referred to as a regular expression, or regex for short.Manipulating Text in LinuxRemember, nearly everything in Linux is a file, and for that matter, most are simple text files. Unlike Windows, with elaborate snap-ins and MMCs to configure an application or server, Linux simply has a text file for configuration. Change the text file, change the configuration. As a result, early pioneers in Linux developed some rather elaborate and elegant ways to manipulate text.We've looked at a few simple ways to manipulate text already, such asgrep and sed, but with regex we'll have the capability to find much more complex text patterns.For instance, what if we we're looking for a line of code among millions of lines of code that began with an "s" containing only the letters "sugr" and the numbers 1-5 with a "bb" ending? Could we find it without having to go through millions of lines of code? Yes—with regex!The Importance of Learning RegexRegex is implemented throughout the information technology world. First developed in 1956 and adopted by Ken Thompson in the original Unix, it has now found its way into Java, Ruby, PHP, Perl, Python, MySQL, Apache, .NET, and of course, Linux.Without understanding regex, you're not only hamstrung in scripting any of these languages, but your ability to do more than simple search and replaces becomes very tedious. In addition, many of the rules written intoSnortand other intrusion detection systems are written in regex.As you can imagine, if searching for some malicious code, the ability to search and find sophisticated and complex text patterns is crucial.How Regex Works in a Security EnvironmentIn this Null Byte tutorial, we'll be using examples fromthe Snort rulesetto illuminate how regex works in a hacking/security environment.Step 1: A Snort RuleOf the many applications and scripting languages that use regular expressions, Snort is one. With its ability to detect just about any type of attack, Snort would be crippled without its regex capabilities. Let's look at new rule that came out just few weeks ago to detect the Ransomware attacks that were seen across the world.The Snort Rule for Detecting Ransomware Attacksalert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"MALWARE-CNC Win.Ransomware.PRISM outbound connection attempt - Get lock screen"; flow:to_server,established; content:"GET"; http_method; content:"/page/index_htm_files2/"; nocase; fast_pattern:only;pcre:"/\x2f((xr)_a-z)|[0-9]{3,}\x2e(css|js|jpg|png|txt)$/U";http_uri; metadata:impact_flag red, policy balanced-ips drop, policy security-ips drop, ruleset community, service http; reference:url,http://www.virustotal.com/en/file/417cb84f48d20120b92530c489e9c3ee9a9deab53fddc0dc153f1034d3c52c58/analysis/1377785686/; classtype:trojan-activity; sid:1000033; rev:3;)End RuleNote the section that is set off and in bold. This is the part of the rule that is utilizingpcreto detect the ransomware.We'll come back to this rule in a later tutorial, but for now, let's look at a simple Snort rule using regular expressions. If you are unfamiliar with Snort rules, make sure to check out my previous guide onreading and writing Snort rules.For our example, let's use this following pseudo-rule:alert tcp any any -> any 80 ( pcre:"/\/foo.php?id=[0-9]{1,10}/";)The first part of this rule should be familiar to us. It says "send analertwhen a packet comes across the wire using thetcpprotocol fromanyIP address fromanyport toanyIP address to port80". It's what comes after the header of this rule that is new and strange.Our task now, is to figure out what this rule is looking for.Step 2: Some Basic SyntaxBefore we begin to attempt to decipher what that rule is looking for, let's lay out basic and simple regular expression syntax and rules./- Begins and ends a regular expression..- Matches any single character.[ ]- Matches a single character within the brackets.[^]- Matches everything except what is in-between the brackets (and after the^).[x-y]Matches every character or number in-between m & n (ex:[a-d]will match the letters a,b,c, or d and[2-7]will match the numbers 2,3,4,5,6, and 7. They are case sensitive by default, and can be combined however you like. For example, to match any alphanumeric character, you can use[A-Za-z0-9]).^- Matches the starting position of the string.* - Matches the preceding element or group zero or more times.$- Matches the ending position of the string.( )- Defines an expression or group.{n}- Matches the preceding character n times (ex:{5}would require the preceding character or group to match 5 times).{m,n}- Matches the preceding element at least m times and not more than n times (ex:{2,4}would require the preceding character or group to appear 2-4 times in a row).|- Matches the character or group either before OR after the|.Step 3: Interpreting the RuleThat summarizes some of the very basic rules of regular expressions. Let's try breaking down the regular expression built into the Snort rule above and try to determine what it is looking for.pcre:"/\/foo.php?id=[0-9]{1,10}/";pcre:- This simply tells the Snort engine to start using Perl Compatible Regular expressions on everything that follows."- Indicates the beginning of the content./- Indicates the beginning of the subexpressions that the PCRE is looking for.\- This is an escape character—it says "don't use the special meaning that the following character has in pcre," but instead see it as literal character./foo.php?id=- This is simple text—the rule is looking for this set of characters.[0-9]- The brackets here indicate look for any of the digits between 0-9.{1,10}- The curly braces here say look for the previous digits between 1 and 10 times./- End the expression we are searching for.We could then interpret this rule in English to say, "look for (presumably a URL) that end with "foo.php?id=" and then has a single digit between 0 and 9 and that digit can be repeated between 1 and 10 times."This rule would then catch packets with:foo.php?id=1foo.php?id=3foo.php?id=33foo.php?id=333333But would pass packets with:bar.php?id=1bar instead of foofoo.php?id=must have at least one digitfoo.php?id=Amust have a digit not an alphabeticfoo.php?id=11111111111can only have between 1 and 10 digits after the =That completes our introduction to regular expressions, but we will return to this subject soon to delve a bit deeper into this powerful but cryptic language, so that you will able to read regex as fluently as you read this page.Want to start making money as a white hat hacker?Jump-start your hacking career with our2020 Premium Ethical Hacking Certification Training Bundlefrom the newNull Byte Shopand get over 60 hours of training from cybersecurity professionals.Buy Now (90% off) >Other worthwhile deals to check out:97% off The Ultimate 2021 White Hat Hacker Certification Bundle99% off The 2021 All-in-One Data Scientist Mega Bundle98% off The 2021 Premium Learn To Code Certification Bundle62% off MindMaster Mind Mapping Software: Perpetual LicenseRelatedHack Like a Pro:Perl Scripting for the Aspiring Hacker, Part 1Hack Like a Pro:An Introduction to Regular Expressions, Part 2News:The Best Cyber Monday Tech Deals on TVs, Phones, Laptops, & MoreHow To:Take Slo-Mo Selfies (AKA 'Slofies') on iPhone 11, 11 Pro & 11 Pro MaxHow To:Create This Easy Hologram Projector for Your Smartphone in Just Ten MinutesNews:Jeep's New AR Experience Lets You Interact with a Car That Isn't Actually ThereHow To:Earn the "Sally Likes Blood" achievement in Call of Duty: Black OpsHow To:Draw a Vampire-SmirkHow To:Abe Lincoln Vampire HunterHow To:Get started using FileMaker Pro 11How To:9 Oreo Hacks That Make Life Worth LivingHow To:Use Colorista plugin in Final Cut Pro (Introduction)How To:Force Restart Your iPhone SE (2nd Generation) When It's Frozen or Otherwise Acting UpNews:iOS 12.1.1 Public Beta 3 Now Available to Software TestersNews:Amazon's Best Black Friday Tech Deals 2016—TVs, Phones & TabletsHow To:Expand Your Coding Skill Set with This 10-Course Training BundleHow To:Get LED Color Effects for Music Playing on Your AndroidNews:You Can Get a OnePlus 7 Pro with INSANE Specs for the Price of theXRor S10eHow To:Get All the New iPad Pro Wallpapers on Your iPhoneHow To:This Quick & Easy Shortcut Minces Garlic in SecondsHow To:You Can Easily Hack Instagram for a Crazy Amount of Likes (But You Totally Shouldn't)How To:Find All the Reels You Liked & Saved on InstagramHow To:Save Custom Shooting Presets in Filmic Pro So You Don't Have to Adjust Settings Later for Similar ShotsHack Like a Pro:How to Install BackTrack 5 (With Metasploit) as a Dual Boot Hacking SystemHow To:Use the guitar amp effect in Pro ToolsHow To:Export video to the web with Final Cut ProHow To:Use Regular Expressions to find text patternsNews:Game box art - what to regular people think?News:introductionHow To:The Official Google+ Insider's Guide IndexNews:Welcome to Edit on a DimeNull Byte:Never Let Us DieNews:Quiksilver Pro El Salvador:How To:Hack Your Old Printer to Automatically Print Hashtag-Based Instagram PhotosNews:Bricked iPad Pros, Error 56, & How to Fix