Repository: caddyserver/caddy (language: go)

Issue:
[RFE] filter/split access logs by fields other than hostname
Hi!

Currently, I can create an access log like so:

```
    log {
        output file /opt/servers/server1/logs/caddy.access.log
    }
```

However, with many health checks coming in, I would redirect all known User-Agents for health checks to its own log file:

```
    log access_log {
        filter not @health-user-agents && not @health-url-paths
        output file /opt/servers/server1/logs/caddy.access.log
    }
    log health_check_log {
        filter @health-user-agents || @health-url-paths
        output file /opt/servers/server1/logs/caddy.access.health.log
    }
```

Of course this is pseudo syntax, but I think you get the idea.

For reference, in Apache httpd I can write something like this:

```
SetEnvIf Request_URI ".*\/health-check$" HealthPattern
# just add more here, they are or'd

CustomLog     /opt/servers/server1/logs/caddy.access.health.log    combined    env=HealthPattern
# and for access.log:  env!HealthPattern
```


Discussion / hints:
Interesting.

I can see a way we could do this. We'd need to adjust https://github.com/caddyserver/caddy/blob/7d919af01b31aff6eb1086e87784bf59c52419bb/modules/caddyhttp/logging.go#L61 to take the whole Request instead of just the host, then we could use `GetVar()` to get a special variable like `access_logger_name` or something like that which is an override for the logger name for the current context. The user could do something like:

```
@healthCheckLog `header_regexp('User-Agent': '^some-regexp$') || path('/healthz*')`
vars @healthCheckLog access_logger_name "health_check_log"
```

PRs welcome if you want to take a crack at it.
Yup, exactly. For me, at least, it is not really high priority. And I would probably see whether other users +1'd this idea.
@francislavoie Is this still up for grabs? I would like to take a crack at it. 
@armadi1809 yeah, still up for grabs.

Thing is though, I'm not 100% convinced that doing this is a good idea because it makes an already complex logging system even more complex. This functionality might be hard to document and explain.

Anyway this is sorta what I was envisioning (run `caddy adapt -p` on this config to see the adapted JSON, important for discussion here):

```
example.com {
	log health_check_log {
		output file /var/logs/caddy.access.health.log
	}
	log {
		output file /var/logs/caddy.access.log
	}

	@healthCheck `header_regexp('User-Agent': '^some-regexp$') || path('/healthz*')`
	handle @healthCheck {
		vars access_logger_name health_check_log
		respond "Healthy"
	}

	handle {
		respond "Hello World"
	}
}
```

This Caddyfile :point_up: produces sensible JSON config. The logger names only has `log0` (the 2nd `log` without a name), and with the change in behaviour I mentioned earlier we could use that `access_logger_name` variable to change which logger gets written to for a given request.

The problem is that if you order the `log` blocks in the opposite order (the one with no name -> `log0`, and then `health_check_log`), then `logger_names` will have `health_check_log` set as the logger for all logs (because the last logger name for a given hostname wins). This is a foot-gun, because order of `log` in the config would matter. I don't like this.

We'd need to think of a way to tell the Caddyfile adapter to ignore a given logger (i.e. `health_check_log`) from being used for `logger_names`. This probably means adding a subdirective like `no_hostname` or something (name TBD) which excludes it so that `logger_names` always has `log0`. I can't think of a way for us to do this automatically in a way that doesn't break other usecases.

Also I took a look a the code, we'll need to move around the call to `s.Logs.wrapLogger` for the error routes, because that gets called before we execute the routes, it should be called after so that the request with the `vars` populated can be used, otherwise `vars` won't be filled yet.
That makes sense. I guess I will start the implementation and try to keep you in the loop. We'll then decide if we end up rolling with it or not. Thank you! 
Thanks! Even if you did not implement it, no blocker for me.  Just thought it might be useful for others as well.

Sorting out load balancer requests etc. Makes log files easier to browse locally. That's all.

Fix patch:
diff --git a/caddyconfig/httpcaddyfile/builtins.go b/caddyconfig/httpcaddyfile/builtins.go
index 27ab0a1826a..35a08ef2782 100644
--- a/caddyconfig/httpcaddyfile/builtins.go
+++ b/caddyconfig/httpcaddyfile/builtins.go
@@ -51,6 +51,7 @@ func init() {
 	RegisterDirective("log", parseLog)
 	RegisterHandlerDirective("skip_log", parseLogSkip)
 	RegisterHandlerDirective("log_skip", parseLogSkip)
+	RegisterHandlerDirective("log_name", parseLogName)
 }
 
 // parseBind parses the bind directive. Syntax:
@@ -914,7 +915,7 @@ func parseLogHelper(h Helper, globalLogNames map[string]struct{}) ([]ConfigValue
 	// this is useful for setting up loggers per subdomain in a site block
 	// with a wildcard domain
 	customHostnames := []string{}
-
+	noHostname := false
 	for h.NextBlock(0) {
 		switch h.Val() {
 		case "hostnames":
@@ -1000,6 +1001,12 @@ func parseLogHelper(h Helper, globalLogNames map[string]struct{}) ([]ConfigValue
 				cl.Exclude = append(cl.Exclude, h.Val())
 			}
 
+		case "no_hostname":
+			if h.NextArg() {
+				return nil, h.ArgErr()
+			}
+			noHostname = true
+
 		default:
 			return nil, h.Errf("unrecognized subdirective: %s", h.Val())
 		}
@@ -1007,7 +1014,7 @@ func parseLogHelper(h Helper, globalLogNames map[string]struct{}) ([]ConfigValue
 
 	var val namedCustomLog
 	val.hostnames = customHostnames
-
+	val.noHostname = noHostname
 	isEmptyConfig := reflect.DeepEqual(cl, new(caddy.CustomLog))
 
 	// Skip handling of empty logging configs
@@ -1058,3 +1065,13 @@ func parseLogSkip(h Helper) (caddyhttp.MiddlewareHandler, error) {
 	}
 	return caddyhttp.VarsMiddleware{"log_skip": true}, nil
 }
+
+// parseLogName parses the log_name directive. Syntax:
+//
+//	log_name <names...>
+func parseLogName(h Helper) (caddyhttp.MiddlewareHandler, error) {
+	h.Next() // consume directive name
+	return caddyhttp.VarsMiddleware{
+		caddyhttp.AccessLoggerNameVarKey: h.RemainingArgs(),
+	}, nil
+}
diff --git a/caddyconfig/httpcaddyfile/directives.go b/caddyconfig/httpcaddyfile/directives.go
index 17259421800..3e688ebcf0f 100644
--- a/caddyconfig/httpcaddyfile/directives.go
+++ b/caddyconfig/httpcaddyfile/directives.go
@@ -53,6 +53,7 @@ var defaultDirectiveOrder = []string{
 	"log_append",
 	"skip_log", // TODO: deprecated, renamed to log_skip
 	"log_skip",
+	"log_name",
 
 	"header",
 	"copy_response_headers", // only in reverse_proxy's handle_response
diff --git a/caddyconfig/httpcaddyfile/httptype.go b/caddyconfig/httpcaddyfile/httptype.go
index 8e7d21fa04b..a8a2ae5b3c5 100644
--- a/caddyconfig/httpcaddyfile/httptype.go
+++ b/caddyconfig/httpcaddyfile/httptype.go
@@ -797,6 +797,15 @@ func (st *ServerType) serversFromPairings(
 			sblockLogHosts := sblock.hostsFromKeys(true)
 			for _, cval := range sblock.pile["custom_log"] {
 				ncl := cval.Value.(namedCustomLog)
+
+				// if `no_hostname` is set, then this logger will not
+				// be associated with any of the site block's hostnames,
+				// and only be usable via the `log_name` directive
+				// or the `access_logger_names` variable
+				if ncl.noHostname {
+					continue
+				}
+
 				if sblock.hasHostCatchAllKey() && len(ncl.hostnames) == 0 {
 					// all requests for hosts not able to be listed should use
 					// this log because it's a catch-all-hosts server block
@@ -1598,9 +1607,10 @@ func (c counter) nextGroup() string {
 }
 
 type namedCustomLog struct {
-	name      string
-	hostnames []string
-	log       *caddy.CustomLog
+	name       string
+	hostnames  []string
+	log        *caddy.CustomLog
+	noHostname bool
 }
 
 // sbAddrAssociation is a mapping from a list of
diff --git a/modules/caddyhttp/logging.go b/modules/caddyhttp/logging.go
index 774bd22f54d..1fab1e8e830 100644
--- a/modules/caddyhttp/logging.go
+++ b/modules/caddyhttp/logging.go
@@ -69,15 +69,35 @@ type ServerLogConfig struct {
 
 // wrapLogger wraps logger in one or more logger named
 // according to user preferences for the given host.
-func (slc ServerLogConfig) wrapLogger(logger *zap.Logger, host string) []*zap.Logger {
-	// logger config should always be only
-	// the hostname, without the port
-	hostWithoutPort, _, err := net.SplitHostPort(host)
+func (slc ServerLogConfig) wrapLogger(logger *zap.Logger, req *http.Request) []*zap.Logger {
+	// using the `log_name` directive or the `access_logger_names` variable,
+	// the logger names can be overridden for the current request
+	if names := GetVar(req.Context(), AccessLoggerNameVarKey); names != nil {
+		if namesSlice, ok := names.([]any); ok {
+			loggers := make([]*zap.Logger, 0, len(namesSlice))
+			for _, loggerName := range namesSlice {
+				// no name, use the default logger
+				if loggerName == "" {
+					loggers = append(loggers, logger)
+					continue
+				}
+				// make a logger with the given name
+				loggers = append(loggers, logger.Named(loggerName.(string)))
+			}
+			return loggers
+		}
+	}
+
+	// get the hostname from the request, with the port number stripped
+	host, _, err := net.SplitHostPort(req.Host)
 	if err != nil {
-		hostWithoutPort = host
+		host = req.Host
 	}
 
-	hosts := slc.getLoggerHosts(hostWithoutPort)
+	// get the logger names for this host from the config
+	hosts := slc.getLoggerHosts(host)
+
+	// make a list of named loggers, or the default logger
 	loggers := make([]*zap.Logger, 0, len(hosts))
 	for _, loggerName := range hosts {
 		// no name, use the default logger
@@ -85,6 +105,7 @@ func (slc ServerLogConfig) wrapLogger(logger *zap.Logger, host string) []*zap.Lo
 			loggers = append(loggers, logger)
 			continue
 		}
+		// make a logger with the given name
 		loggers = append(loggers, logger.Named(loggerName))
 	}
 	return loggers
@@ -211,4 +232,7 @@ const (
 
 	// For adding additional fields to the access logs
 	ExtraLogFieldsCtxKey caddy.CtxKey = "extra_log_fields"
+
+	// Variable name used to indicate the logger to be used
+	AccessLoggerNameVarKey string = "access_logger_names"
 )
diff --git a/modules/caddyhttp/server.go b/modules/caddyhttp/server.go
index 408f3358c80..1d621456d50 100644
--- a/modules/caddyhttp/server.go
+++ b/modules/caddyhttp/server.go
@@ -369,7 +369,7 @@ func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 	errLog = errLog.With(zap.Duration("duration", duration))
 	errLoggers := []*zap.Logger{errLog}
 	if s.Logs != nil {
-		errLoggers = s.Logs.wrapLogger(errLog, r.Host)
+		errLoggers = s.Logs.wrapLogger(errLog, r)
 	}
 
 	// get the values that will be used to log the error
@@ -778,7 +778,7 @@ func (s *Server) logRequest(
 
 	loggers := []*zap.Logger{accLog}
 	if s.Logs != nil {
-		loggers = s.Logs.wrapLogger(accLog, r.Host)
+		loggers = s.Logs.wrapLogger(accLog, r)
 	}
 
 	// wrapping may return multiple loggers, so we log to all of them
@@ -835,7 +835,6 @@ func PrepareRequest(r *http.Request, repl *caddy.Replacer, w http.ResponseWriter
 	ctx = context.WithValue(ctx, OriginalRequestCtxKey, originalRequest(r, &url2))
 
 	ctx = context.WithValue(ctx, ExtraLogFieldsCtxKey, new(ExtraLogFields))
-
 	r = r.WithContext(ctx)
 
 	// once the pointer to the request won't change


Test patch:
diff --git a/caddytest/integration/caddyfile_adapt/log_filter_with_header.txt b/caddytest/integration/caddyfile_adapt/log_filter_with_header.txt
new file mode 100644
index 00000000000..3ab6d6246cf
--- /dev/null
+++ b/caddytest/integration/caddyfile_adapt/log_filter_with_header.txt
@@ -0,0 +1,151 @@
+localhost {
+	log {
+		output file ./caddy.access.log
+	}
+	log health_check_log {
+		output file ./caddy.access.health.log
+		no_hostname
+	}
+	log general_log {
+		output file ./caddy.access.general.log
+		no_hostname
+	}
+	@healthCheck `header_regexp('User-Agent', '^some-regexp$') || path('/healthz*')`
+	handle @healthCheck {
+		log_name health_check_log general_log
+		respond "Healthy"
+	}
+
+	handle {
+		respond "Hello World"
+	}
+}
+----------
+{
+	"logging": {
+		"logs": {
+			"default": {
+				"exclude": [
+					"http.log.access.general_log",
+					"http.log.access.health_check_log",
+					"http.log.access.log0"
+				]
+			},
+			"general_log": {
+				"writer": {
+					"filename": "./caddy.access.general.log",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.general_log"
+				]
+			},
+			"health_check_log": {
+				"writer": {
+					"filename": "./caddy.access.health.log",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.health_check_log"
+				]
+			},
+			"log0": {
+				"writer": {
+					"filename": "./caddy.access.log",
+					"output": "file"
+				},
+				"include": [
+					"http.log.access.log0"
+				]
+			}
+		}
+	},
+	"apps": {
+		"http": {
+			"servers": {
+				"srv0": {
+					"listen": [
+						":443"
+					],
+					"routes": [
+						{
+							"match": [
+								{
+									"host": [
+										"localhost"
+									]
+								}
+							],
+							"handle": [
+								{
+									"handler": "subroute",
+									"routes": [
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"access_logger_names": [
+																		"health_check_log",
+																		"general_log"
+																	],
+																	"handler": "vars"
+																},
+																{
+																	"body": "Healthy",
+																	"handler": "static_response"
+																}
+															]
+														}
+													]
+												}
+											],
+											"match": [
+												{
+													"expression": {
+														"expr": "header_regexp('User-Agent', '^some-regexp$') || path('/healthz*')",
+														"name": "healthCheck"
+													}
+												}
+											]
+										},
+										{
+											"group": "group2",
+											"handle": [
+												{
+													"handler": "subroute",
+													"routes": [
+														{
+															"handle": [
+																{
+																	"body": "Hello World",
+																	"handler": "static_response"
+																}
+															]
+														}
+													]
+												}
+											]
+										}
+									]
+								}
+							],
+							"terminal": true
+						}
+					],
+					"logs": {
+						"logger_names": {
+							"localhost": [
+								"log0"
+							]
+						}
+					}
+				}
+			}
+		}
+	}
+}