SpiderLabs/ModSecurity — pull request #723 by defanator (2014-05-19 13:02:00+00:00)
Title: A couple of changes to fix header and body filter processing

Hi @zimmerle,

these commits fixes almost all tests for core nginx functionality (i.e., for all modules built by default - I've not tested optional modules yet) included in http://hg.nginx.org/nginx-tests:

https://github.com/defanator/ModSecurity/commit/77feb2b76112de3113b721778f92925600280eb1
https://github.com/defanator/ModSecurity/commit/a20c7a70917a6d2157273705bb9bd4eb546025b4

The only one failed test remained is "handshake pipelined" from proxy_upgrade.t.

I've used the following patch against nginx-tests repository to simplify testing:

diff -r 25c9cd16ae04 lib/Test/Nginx.pm
--- a/lib/Test/Nginx.pm	Fri May 16 21:19:21 2014 +0400
+++ b/lib/Test/Nginx.pm	Mon May 19 10:18:17 2014 +0400
@@ -25,6 +25,7 @@
use POSIX qw/ waitpid WNOHANG /;
use Socket qw/ CRLF /;
use Test::More qw//;
+use File::Copy qw(copy);

###############################################################################

@@ -46,6 +47,14 @@
mkdir "$self->{_testdir}/logs"
or die "Can't create logs directory: $!\n";

+	if ($ENV{TEST_NGINX_MODSECURITY_CONF_TEMPLATE}) {
+	my $c = `cat $ENV{TEST_NGINX_MODSECURITY_CONF_TEMPLATE}`;
+	$self->write_file_expand('mod_security.conf', $c);
+	}
+
+	copy $0, $self->{_testdir} . '/';
+	chdir $self->{_testdir};
+
return $self;
}

@@ -393,6 +402,9 @@
$s .= "scgi_temp_path $self->{_testdir}/scgi_temp;\n"
if $self->has_module('scgi');

+	$s .= "ModSecurityEnabled on;\nModSecurityConfig mod_security.conf;\n"
+	if $ENV{TEST_NGINX_MODSECURITY_CONF_TEMPLATE};
+
$s .= $ENV{TEST_NGINX_GLOBALS_HTTP}
if $ENV{TEST_NGINX_GLOBALS_HTTP};

In order to get it work, the following environment variables should be set:

TEST_NGINX_BINARY=/home/defan/src/nginx-1.7.0/objs/nginx
TEST_NGINX_LEAVE=yes
TEST_NGINX_MODSECURITY_CONF_TEMPLATE=/home/defan/modsec.conf

Here's my modsec.conf template:

SecRuleEngine DetectionOnly
SecRequestBodyAccess On
SecRule REQUEST_HEADERS:Content-Type "text/xml" \
"id:'200000',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"
SecRule REQUEST_HEADERS:Content-Type "application/json" \
"id:'200001',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"
SecRequestBodyLimit 13107200
SecRequestBodyNoFilesLimit 131072
SecRequestBodyInMemoryLimit 131072
SecRequestBodyLimitAction Reject
SecRule REQBODY_ERROR "!@eq 0" \
"id:'200002', phase:2,t:none,log,deny,status:400,msg:'Failed to parse request body.',logdata:'%{reqbody_error_msg}',severity:2"
SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
"id:'200003',phase:2,t:none,log,deny,status:400, \
msg:'Multipart request body failed strict validation: \
PE %{REQBODY_PROCESSOR_ERROR}, \
BQ %{MULTIPART_BOUNDARY_QUOTED}, \
BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
DB %{MULTIPART_DATA_BEFORE}, \
DA %{MULTIPART_DATA_AFTER}, \
HF %{MULTIPART_HEADER_FOLDING}, \
LF %{MULTIPART_LF_LINE}, \
SM %{MULTIPART_MISSING_SEMICOLON}, \
IQ %{MULTIPART_INVALID_QUOTING}, \
IP %{MULTIPART_INVALID_PART}, \
IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"
SecRule MULTIPART_UNMATCHED_BOUNDARY "!@eq 0" \
"id:'200004',phase:2,t:none,log,deny,msg:'Multipart parser detected a possible unmatched boundary.'"

SecPcreMatchLimit 1000
SecPcreMatchLimitRecursion 1000
SecRule TX:/^MSC_/ "!@streq 0" \
"id:'200005',phase:2,t:none,deny,msg:'ModSecurity internal error flagged: %{MATCHED_VAR_NAME}'"

SecResponseBodyAccess On
SecResponseBodyMimeType text/plain text/html text/xml
SecResponseBodyLimit 524288
SecResponseBodyLimitAction ProcessPartial

SecTmpDir /tmp/
SecDataDir /tmp/

SecDebugLog %%TESTDIR%%/modsec_debug.log
SecDebugLogLevel 9

SecAuditEngine RelevantOnly
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLogParts ABIJDEFHZ
SecAuditLogType Serial
SecAuditLog %%TESTDIR%%/modsec_audit.log

SecArgumentSeparator &

SecCookieFormat 0

SecStatusEngine On

PS: sorry for mixed-up pull requests, I still can't figure out how to deal with selective pull requests including a limited set of commits.