Question: How do I troubleshoot issues with Network Firewall when a rule isn't working as expected?

I want to troubleshoot issues with AWS Network Firewall when a rule isn't working as expected.

Short descriptionThe following scenarios can cause Network Firewall rules not to work as expected:Traffic isn't symmetrically routed through the firewall.The Network Firewall rule is incorrectly configured.Note: Before you troubleshoot Network Firewall issues, confirm the following configurations:Firewall endpoints are deployed in their dedicated subnets and route tables. The firewall subnets must not have any workload resources.Route tables route traffic through the firewall endpoint. Check the Amazon CloudWatch metric ReceivedPackets to verify that the Network Firewall received packets. If the ReceivedPackets metric has a value of zero, then verify that the route tables have a correct route that points to the firewall endpoint.

ResolutionNote: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.Traffic isn't symmetrically routed through the firewallFor a centralized Network Firewall deployment model, turn on appliance mode for the transit gateway attachment of the inspection Virtual Private Cloud (VPC).Run the following AWS CLI command to check if appliance mode is turned on:aws ec2 describe-transit-gateway-vpc-attachments --filters Name=transit-gateway-attachment-id,Values=tgw-attach-exampleRun the following AWS CLI command to turn on appliance mode:aws ec2 modify-transit-gateway-vpc-attachment --transit-gateway-attachment-id tgw-attach-example --options ApplianceModeSupport="enable"Note: In the preceding example commands, replace tgw-attach-example with the transit gateway attachment of the inspection VPC.When deploying a firewall endpoint in a public subnet, the internet gateway's ingress route table must route to the private subnet through the firewall endpoint. The private subnet's route table must also route internet traffic to the same firewall endpoint. For more information, see Simple single zone architecture with an internet gateway.For a Multi-AZ firewall deployment setup, the internet gateway's ingress route table must route traffic to the private subnets through the same firewall endpoint. The route table must also be in the same Availability Zone as the private subnet. For more information, see Multi zone architecture with an internet gateway.The Network Firewall rule is incorrectly configuredIssues with Network Firewall rules might be due to how you configure the rule.Stateless rulesFor traffic to flow in both directions, you must apply stateless rules to the ingress and egress rules. If you apply a rule on the ingress for incoming traffic, then you must also apply a rule to the egress for the response traffic. If you apply a rule to the egress for the response traffic, then you must also apply a rule on the ingress for incoming traffic. This rule configuration is similar to an access control list (ACL).The following example shows how to use stateless rules to allow incoming SSH traffic.PriorityProtocolSourceDestinationSource port rangeDestination port rangeAction1TCP0.0.0.0/010.1.0.0/160:6553522Pass2TCP10.1.0.0/160.0.0.0/0220:65535Pass Note: In the preceding example, incoming SSH traffic is routed through the CIDR 10.1.0.0/16 from any IP address.Stateless default actions for packetsReview the stateless default actions for both full packets and fragmented packets. The default action settings determine how the firewall stateless engine handles individual packets when they don't match the stateless rules. Apply the stateless default action for fragmented packets to only IPv4 packet fragments that use the UDP transport protocol.Note: The IPv6 protocol doesn't support fragmentation in the network. If a host sends a packet that’s larger than the receiving host’s MTU, then the receiving host drops the packet. Devices on the path also drop packets that are larger than the device’s MTU.Stateful rulesFor the stateful rule engine to inspect traffic, the stateless rule engine must forward both traffic flow directions to stateful rule groups.Network Firewall rules must refer to private IP addressesIf a private subnet or workload has both public and private IP addresses, then the Network Firewall rule must refer to the private IP address.Rule group variablesStateful Network Firewall rule groups use the variable HOME_NET to define the scope of the firewall inspection. If you don't explicitly specify the HOME_NET variable, then the variable defaults to the VPC CIDR range where the Network Firewall is deployed. In this default setting, the Network Firewall inspects traffic that originates from the sources within the VPC.When deploying a centralized Network Firewall, the HOME_NET variable of the rule group must include your remote VPC. Or, it must include the source CIDRs where your traffic is originating from. For more information, see Stateful domain list rule groups in AWS Network Firewall.To inspect traffic from only certain private subnets and not the entire VPC, the HOME_NET variable must contain only the source subnet CIDRs. In the following example, the HOME_NET variable contains the source subnet CIDRs:"RuleVariables": {        "IPSets": {           "HOME_NET": {             "Definition": [               "10.0.0.0/16",               "10.1.0.0/16",               "192.168.2.0/24"             ]           }        }    }You can use the DescribeRuleGroup API to check the stateful rule group details. If the RuleVariables object isn't in the response, then the firewall rule group uses default settings. The default settings for HOME_NET are set to the Network Firewall VPC CIDR.Rule evaluation orderApplications that use TCP require the 3-way handshake to be successful before the application requests are sent. The first packet that the firewall sees in the connection is a TCP SYN from the source. If conflicting rules block the lower-level TCP packets, then the 3-way handshake doesn't succeed and causes the application to time out.For example, to detect the hostname, a domain list rule group uses the Host header in the HTTP request. It also uses the Server Name Indication (SNI) extension in the TLS handshake. For an HTTP request and TLS negotiation to happen, the initial 3-way TCP handshake on ports 80 (HTTP) and 443 (HTTPS) must first complete.When you use rules that involve both application and transport layers, make sure to allow traffic on the TCP ports that the application requires. Review the rule evaluation order to verify that the preceding rules match the traffic.With default action order, make sure that other rules aren't blocking traffic during evaluation. Use the flow keyword in your rules to avoid matching against lower-level protocol packets before higher-level application protocols are identified.Rule configuration examplesIncorrect rule configuration for TCP trafficIn the following example, the rule is incorrectly configured to block all outbound TCP connections and allow only HTTP requests to example.com. Because all the TCP traffic is dropped, the source client can't send the HTTP application request.pass http $HOME_NET any -> $EXTERNAL_NET 80 (http.host; dotprefix; content:"example.com"; endswith; msg:"Allowed HTTP domain"; sid:172191; rev:1;) drop tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"Drop outbound TCP traffic"; sid:172192; rev:1;)Correct configuration of TCP connectionIn the following example, the Drop established TCP:80 rule establishes the TCP connection first. Then, it drops all further packets that don't match the HTTP Hostname example.com.drop tcp $HOME_NET any -> $EXTERNAL_NET 80 (msg:"Drop established TCP:80"; flow: from_client,established; sid:172190; rev:1;)pass http $HOME_NET any -> $EXTERNAL_NET 80 (http.host; dotprefix; content:"example.com"; endswith; msg:"Allowed HTTP domain"; priority:10; sid:172191; rev:1;)drop tcp $HOME_NET any -> $EXTERNAL_NET !80 (msg:"Drop outbound non TCP:80 traffic"; sid:172192; rev:1;)Note: Default action order automatically allows the packets that don't match any defined rules.With strict evaluation order, rule groups are evaluated in order of priority, starting with the lowest number. The rules in each rule group are processed in the order that they're defined. Make sure that the configurations for the rules and manual priority order are correct. Don't match a lower priority rule against your network traffic. Also, write the rules based on the default action that you select on the strict firewall policy.Incorrect configuration causing timeoutsIn the following example, the specified configuration drops all the TCP traffic by default before the firewall can detect the HTTP application-level request. This configuration causes unexpected timeouts.Default Action: Drop allRule: pass http $HOME_NET any -> $EXTERNAL_NET 80 (http.host; dotprefix; content:"example.com"; endswith; msg:"Allowed HTTP domain"; sid:172191; rev:1;)Correct configuration allowing TCP 3-way handshakeIn the following example, the Network Firewall allows the TCP 3-way handshake to be established and allows only HTTP traffic for the domain example.com.Default Action: Drop establishedRule: pass http $HOME_NET any -> $EXTERNAL_NET 80 (http.host; dotprefix; content:"example.com"; endswith; msg:"Allowed HTTP domain"; sid:172191; rev:1;)For more firewall rule examples, see Examples of stateful rules for Network Firewall.

Related informationDeployment models for AWS Network FirewallDefining rule actions in AWS Network FirewallDomain list inspection for traffic from outside the deployment VPCEvaluation order for stateful rule groups