Remove the allow/deny stuff