Repository: caddyserver/caddy (language: go)

Issue:
Constant reloading of configfile in 2.7.4
Just upgraded from caddy 2.5-ish I guess, to 2.7.4.

I get the constant reloaded from config file as well now.
Dumping the autosave.json through `jq` and to different files and then looking at the differences,
it appears there is another 'undeterministic' problem in the vhost names.

```diff
*** 1.json      2023-08-29 15:27:53.476468912 +0200
--- 2.json      2023-08-29 15:28:01.206467772 +0200
***************
*** 3081,3089 ****
                },
                "match": {
                  "sni": [
                    "thiccrm.dev.local",
!                   "thicform.dev.local",
!                   "thicpartners.dev.local"
                  ]
                }
              },
--- 3081,3089 ----
                },
                "match": {
                  "sni": [
+                   "thicpartners.dev.local",
                    "thiccrm.dev.local",
!                   "thicform.dev.local"
                  ]
                }
              },
```

In my caddyfile:
```
thiccrm.dev.local, thicform.dev.local, thicpartners.dev.local {
        bind localhost
        tls /home/joris/vhosts/wildcard.dev.local.crt /home/joris/vhosts/wildcard.dev.local.pem
        root * /home/joris/vhosts/thiccrm/public
        php_fastcgi unix//run/php/php8.1-fpm.sock
        file_server
}
```
(there is a lot more, but this is the culprit)

_Originally posted by @jorismak in https://github.com/caddyserver/caddy/issues/5538#issuecomment-1697452361_
            


Discussion / hints:
Thanks for opening an issue! We'll look into this.

It's not immediately clear to me what is going on, so I'll need your help to understand it better.

Ideally, we need to be able to reproduce the bug _in the most minimal way possible_ **using the latest version of Caddy.** This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! **This will require some effort on your part -- please understand that we will be dedicating time to fix the bug you are reporting if you can just help us understand it and reproduce it easily.**

This template will ask for some information you've already provided; that's OK, just fill it out the best you can. :+1: I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

## Template

<pre><code>## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

This should be the latest version of Caddy:

```
paste here
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)




### 2b. Why it's a bug (if it's not obvious)




### 2c. Log output

```
paste terminal output or logs here
```



### 2d. Workaround(s)




### 2e. Relevant links




## 3. Tutorial (minimal steps to reproduce the bug)




</code></pre>

## Instructions -- please heed otherwise we cannot help you (help us help you!)

1. **Environment**: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are _always_ relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. **Description:** Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.
	- **2c) Log output:** Paste terminal output and/or complete logs in a code block. [DO NOT REDACT INFORMATION except for credentials.](https://caddy.community/t/how-to-get-help-with-caddy-more-effectively/5222) Please enable debug and access logs.
	- **2d) Workaround:** What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
	- **2e) Relevant links:** Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. **Tutorial:** What are the _minimum required specific steps_ someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:
	- Start with an empty config. Add _only_ the lines/parameters that are _absolutely required_ to reproduce the bug.
	- Do not run Caddy inside containers.
	- Run Caddy manually in your terminal; do not use systemd or other init systems.
	- If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like `curl`.
	- Do not redact any information from your config (except credentials). Domain names are public knowledge and [often necessary for quick resolution of an issue](https://caddy.community/t/how-to-get-help-with-caddy-more-effectively/5222)!
	- Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

<blockquote>Create a config file:

```
{ ... }
```

Open terminal and run Caddy:

```
$ caddy ...
```

Make an HTTP request:

```
$ curl ...
```

Notice that the result is ___ but it should be ___.</blockquote>

## 1. Environment

### 1a. Operating system and version

```
Linux amd64 Ubuntu focal (20.04) (Windows WSL)
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

This should be the latest version of Caddy:

```
v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)

Same problem as the mentioned / references issue.

I start caddy with `./caddy start --watch --config /etc/caddy/Caddyfile`.
I get some console output dumped _over_ my current terminal, the log entries for caddy starting up.
Everything is fine.

Wait +/- 3 seconds, and you get almost the same log _again_. And _again_, and _again_, and _again_.

Everytime the logs start with
```
watcher config file changed; reloading  {"config_file": "/etc/caddy/Caddyfile"}
```

The _first_ log dump directly after starting says
```
2023/08/29 14:04:59.348 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2023/08/29 14:04:59.348 INFO    serving initial configuration
Successfully started Caddy (pid=30356) - Caddy is running in the background
root@LAPTOP-G6PO1629:/# 2023/08/29 14:04:59.353 INFO    watcher watching config file for changes        {"config_file": "/etc/caddy/Caddyfile"}
```

So again, everything is fine and working. It just thinks the config file changed every time it checks, while nothing has changed.



### 2e. Relevant links
https://github.com/caddyserver/caddy/issues/5538



## 3. Tutorial (minimal steps to reproduce the bug)

I'm thinking the problem is in my case that in my Caddyfile I have multiple domain names in a single serverblock:

```
thiccrm.dev.local, thicform.dev.local, thicpartners.dev.local {
        root * /home/joris/vhosts/thiccrm/public
        php_fastcgi unix//run/php/php8.1-fpm.sock
        file_server
}
```

What happens is that every time the config file is read and parsed, a ***different*** json output is written to /root/.config/caddy/autosave.json. There are slight changes every time, and this is why Caddy thinkgs the config file is changed constantly.

I quote from the original issue:

> The problem is that maps in Go don't keep items ordered deterministically when iterating through their entries. Slices are ordered through. So the solution will be to sort the Hide slice after filling it with entries from the map.


I'm guessing the exact same problem is happening here with the domain names on a single line in the Caddyfile.
As I've shown in the opening post, the json structure in autosave.json is slightly differently ordered every time in the 'sni' block.

Thanks, we'll look into this as soon as we have a chance!
Wouldn't it be better to check for config file modification time and then load the config
rather than loading the config and checking its content.
https://github.com/caddyserver/caddy/blob/master/cmd/main.go#L189
I don't know if this is going to fix the issue though:
```go
cfgFileInfo, err := os.Stat(filename)
// check err

lastModTime := cfgFileInfo.ModTime()

// load config

// begin poller
for range time.Tick(1 * time.Second) {
	cfgFileInfo, err = os.Stat(filename)
	// check err

	if !cfgFileInfo.ModTime().After(lastModTime) {
		continue
	}
	lastModTime = cfgFileInfo.ModTime()

	// load config
}
```
@amis-shokoohi the problem is a Caddyfile may import other files. We'd need to check the mod time for all files in the tree. We don't have infrastructure for tracking all the files that are imported currently.
There was something with included files and inotify having issues with big
directories and such. That's what I read from the related ticket.

A different implementation for watching maybe on the horizon, maybe not .
But I'd like to have this fixed before they though (but I have nothing to
complain about ofcourse !)

On Mon, Sep 4, 2023, 18:30 Amis Shokoohi ***@***.***> wrote:

> Wouldn't it be better to check for config file modification time and then
> load the config
> rather than loading the config and checking its content.
> https://github.com/caddyserver/caddy/blob/master/cmd/main.go#L189
> I don't know if this is going to fix the issue though:
>
> cfgFileInfo, err := os.Stat(filename)// check err
> lastModTime := cfgFileInfo.ModTime()
> // load config
> // begin pollerfor range time.Tick(1 * time.Second) {
> 	cfgFileInfo, err = os.Stat(filename)
> 	// check err
>
> 	if !cfgFileInfo.ModTime().After(lastModTime) {
> 		continue
> 	}
> 	lastModTime = cfgFileInfo.ModTime()
>
> 	// load config
> }
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/caddyserver/caddy/issues/5789#issuecomment-1705516516>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ALMD2DQOHWMQ42KUXSOMS4LXYX62PANCNFSM6AAAAAA4C66CPU>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

I didn't see this: https://github.com/caddyserver/caddy/pull/5521
It seems that it used to check config file mod time, but as you guys said
other files may be imported in `Caddyfile` as well.
So even if we were able to check all files mod time, we still need to read `Caddyfile`
to check if new files are imported (or maybe some files got deleted).
You know it's just easier to use `loadConfigWithLogger` to be honest. 😅

Fix patch:
diff --git a/caddyconfig/httpcaddyfile/httptype.go b/caddyconfig/httpcaddyfile/httptype.go
index 78fb7f062a1..f6aeb5309f9 100644
--- a/caddyconfig/httpcaddyfile/httptype.go
+++ b/caddyconfig/httpcaddyfile/httptype.go
@@ -685,6 +685,7 @@ func (st *ServerType) serversFromPairings(
 					}
 
 					if len(hosts) > 0 {
+						slices.Sort(hosts) // for deterministic JSON output
 						cp.MatchersRaw = caddy.ModuleMap{
 							"sni": caddyconfig.JSON(hosts, warnings), // make sure to match all hosts, not just auto-HTTPS-qualified ones
 						}


Test patch: