Repository: prowler-cloud/prowler
Pull request: Support whitelists per check
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Fixes https://github.com/toniblyx/prowler/issues/154

cc @dbellizzi 

Problem statement:
Support for resources whitelist (leave resources out of checks)
Would be nice to have environment definable suppressions and ignores:

An example would be to ignore `check73` if `bucket` = `my-known-problem-bucket`.

I am envisioning a file based approach, either one file for all suppressions with a config loader, or potentially a file per check so you can rely on conventions for loading your ignore list.

Example:
```text
# File Name: check73.suppressions
suppress=()
suppress+=('my-known-problem-bucket')
suppress+=('another-problem-bucket')
```

Easier examples would be line separated values or comma perhaps. Lots of options but would be nice.

Hints from issue discussion:
Thanks @endzyme, I think it may be useful for some cases. What is your use case exactly, don't you want to find wrongly configured (deliberately or not) assets? Are you using any kind of automation or CD/CI with Prowler? You can always `grep -v my-known-problem-bucket` to remove the error, I know it seems probably a bit dirty, but enough. Even with a eventual scoring system, you will always know how your account/accounts are in terms of security configuration best practices. What do you think?
Thanks for being so quick to respond. 

Use Case: Some assets aren't misconfigured but are intentionally configured in a certain way that may be against recommendations. An example would be S3 buckets that are public, that could be a business decision that should be ignored.

In terms of `grep`ing our way out of the problem, that seems like a fine MVP. Longer term I think it would be prudent to be able to configure suppressions but I can see how difficult that could be based on all the different ways checks are performed (may require secondary function implementations for filtering, which means probably not bash). 

I am fine closing this for now and using `grep` solution.
S3 use case is very common, we have the same case. I think this can be also addressed if I implement somehow an option to only show `WARN` instead of `OK + WARN`. I'll keep it in mind. Thanks @endzyme 
I've been thinking about this: whitelist! I guess the easier way to implement it is creating a file with all whitelisted devices by its resource id, somethign like `-w whitelist.txt`:
```
check710 i-01234567890123456788
check710 i-01234567890123456789
check75 sg-0123456788
check75 sg-0123456789
```

This would be awesome
The whitelist is probably better than grep as it gives us the option to start building an approved list of resources to ignore.
I had started doing this in an Excel sheet with lookups, which sorta worked. Only feedback on that would be making it so you can easily add a comment to each lined exception.
I've been thinking in a small function that `grep -v` whatever is needed per each check... Can be loaded from a text file per line or something like this:
```
check26:bucket111,bucket222,bucket333
```
 I'm busy with other things until next week, I'll play with it soon. WDYT?
Allowing repeated check numbers might make it easier to comment the file as you go:
```
check26:bucket111 # This is good
check26:bucket222 # Bob did this
check26:bucket333 # This terrible thing is hosting a website
```
I like it, good idea!

Commit message: Support whitelists per check

Patch to review:
diff --git a/include/outputs b/include/outputs
index b18ac68352..c89350b519 100644
--- a/include/outputs
+++ b/include/outputs
@@ -104,15 +104,29 @@ textInfo(){
 }
 
 textFail(){
-  FAIL_COUNTER=$((FAIL_COUNTER+1))
-  EXITCODE=3
+  ## ignore whitelists for current check
+  level="FAIL"
+  for i in $IGNORES; do
+    ignore_value="${i#*${CHECK_NAME}:}"
+    if [[ $1 =~ ${ignore_value} ]]; then
+      level="WARNING"
+      break
+    fi
+  done
+
+  # only set non-0 exit code on FAIL mode, WARN is ok
+  if [[ "$level" == "FAIL" ]]; then
+    FAIL_COUNTER=$((FAIL_COUNTER+1))
+    EXITCODE=3
+  fi
+
   if [[ "$MODE" == "csv" ]]; then
     if [[ $2 ]]; then
       REPREGION=$2
     else
       REPREGION=$REGION
     fi
-    echo "$PROFILE${SEP}$ACCOUNT_NUM${SEP}$REPREGION${SEP}$TITLE_ID${SEP}FAIL${SEP}$ITEM_SCORED${SEP}$ITEM_LEVEL${SEP}$TITLE_TEXT${SEP}$1"
+    echo "$PROFILE${SEP}$ACCOUNT_NUM${SEP}$REPREGION${SEP}$TITLE_ID${SEP}${level}${SEP}$ITEM_SCORED${SEP}$ITEM_LEVEL${SEP}$TITLE_TEXT${SEP}$1"
   elif [[ "$MODE" == "json" ]]; then
     if [[ $2 ]]; then
       REPREGION=$2
@@ -128,7 +142,7 @@ textFail(){
     --arg ITEM_LEVEL "$ITEM_LEVEL" \
     --arg TITLE_ID "$TITLE_ID" \
     --arg REPREGION "$REPREGION" \
-    --arg TIMESTAMP $(date -u +"%Y-%m-%dT%H:%M:%SZ") \
+    --arg TIMESTAMP "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" \
     -n '{
       "Profile": $PROFILE,
       "Account Number": $ACCOUNT_NUM,
@@ -142,7 +156,11 @@ textFail(){
       "Timestamp": $TIMESTAMP,
     }'
   else
-    echo "      $BAD FAIL! $1 $NORMAL"
+    if [[ "${level}" == "FAIL" ]]; then
+      echo "      $BAD ${level}! $1 $NORMAL"
+    else
+      echo "      $WARNING ${level}! $1 $NORMAL"
+    fi
   fi
 }
 
diff --git a/prowler b/prowler
index a3a8db27ce..05ee1e00c6 100755
--- a/prowler
+++ b/prowler
@@ -47,6 +47,7 @@ EXITCODE=0
 SCRIPT_START_TIME=$( date -u +"%Y-%m-%dT%H:%M:%S%z" )
 TITLE_ID=""
 TITLE_TEXT="CALLER ERROR - UNSET TITLE"
+WHITELIST_FILE=""
 
 # Command usage menu
 usage(){
@@ -83,12 +84,19 @@ USAGE:
                             (i.e.: ProwlerRole)
       -T                  session durantion given to that role credentials in seconds, default 1h (3600) recommended 12h, requires -R and -T
                             (i.e.: 43200) 
+      -w                  whitelist file. (Lines starting with # are ignored as comments) Format:
+                            # ignore these due to some reason
+                            # check1 checks s3 buckets
+                            <checkid1>:<resource to ignore 1>
+                            <checkid1>:<resource to ignore 2>
+                            # checkid2
+                            <checkid2>:<resource to ignore 1>
       -h                  this help
   "
   exit
 }
 
-while getopts ":hlLkqp:r:c:g:f:m:M:E:enbVsx:A:R:T:" OPTION; do
+while getopts ":hlLkqp:r:c:g:f:m:M:E:enbVsx:A:R:T:w:" OPTION; do
    case $OPTION in
      h )
         usage
@@ -160,6 +168,11 @@ while getopts ":hlLkqp:r:c:g:f:m:M:E:enbVsx:A:R:T:" OPTION; do
      T )
         SESSION_DURATION_TO_ASSUME=$OPTARG
         ;;
+     w )
+        WHITELIST_FILE=$OPTARG
+        echo ""
+        echo "$OPTNORMAL Using Whitelist file: $OPTARG"
+        ;;
      : )
         echo ""
         echo "$OPTRED ERROR!$OPTNORMAL  -$OPTARG requires an argument"
@@ -202,6 +215,12 @@ REGIONS=$($AWSCLI ec2 describe-regions --query 'Regions[].RegionName' \
   --region $REGION \
   --region-names $FILTERREGION)
 
+# Pre-process whitelist file if supplied
+if [[ -n "$WHITELIST_FILE" ]]; then
+  # ignore lines starting with # (comments)
+  WHITELIST=$(awk   '!/^[[:space:]]*#/{print }' <(cat "$WHITELIST_FILE"))
+fi
+
 # Load all of the groups of checks inside groups folder named as "groupNumber*"
 for group in $(ls $PROWLER_DIR/groups/group[0-9]*|grep -v groupN_sample); do
 	. "$group"
@@ -266,8 +285,10 @@ execute_check() {
           saveReport
         fi
       fi
-		  show_check_title $1
-		  $1
+      show_check_title "$1"
+      ignores=$(awk '/${1}/{print}' <(echo "${WHITELIST}"))
+      # set the custom ignores list for this check
+      IGNORES="${ignores}" $1
     else
       textFail "ERROR! Use a valid check name (i.e. check41 or extra71)";
       exit $EXITCODE
diff --git a/whitelist_sample.txt b/whitelist_sample.txt
new file mode 100644
index 0000000000..e4bccdf85e
--- /dev/null
+++ b/whitelist_sample.txt
@@ -0,0 +1,4 @@
+# Each line is a (checkid:item) tuple
+
+# Example: Will not consider a myignoredbucket failures as full failure. (Still printed as a warning)
+check26:myignoredbucket


Code review comments:
diff_hunk: @@ -83,12 +84,19 @@ USAGE:
                             (i.e.: ProwlerRole)
       -T                  session durantion given to that role credentials in seconds, default 1h (3600) recommended 12h, requires -R and -T
                             (i.e.: 43200) 
+      -w                  whitelist file. (Lines starting with # are ignored as comments) Format:
path: prowler
text: @author:
```file prowler: Line 174 could cause problems if CSV or json is used, am I right? I don 't think it is needed.```

@user1 yeah, I was hoping that this help doc clarifies the structure we're relying on here, unless I am missing something else about the CSV/JSON concern.

@author:
Overall, what do you think about migrating this project to a python/go etc to help simplify some of the abstractions? Just a thought - you might get more contribs and somewhat easier code - I noticed that the shell code in here has a lot of warnings according to shell-check.
It's def your preference as the project owner - hope this doesn't come across the wrong way :)
The checks themselves could stay in bash because that makes using AWS etc apis much easier but the scaffolding around that could be moved and then we could have easy pre-filter, post-filter hooks etc.