Repository: traefik/traefik (language: go)

Issue:
acme.json in an invalid JSON, Traefik can't read certs
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you do?

I use Traefik as a Docker container on several servers, with both the Docker Provider and the File Provider working side by side (the File Provider will soon be deprecated). 

Each Traefik container handles about 100 services (websites in their own Docker containers). I use the HTTP challenge.

My configuration works fine, and generates certificates accordingly. The certs are written to the `acme.json` file automatically by Traefik. This is where the problem lies. 

### What did you see instead?

For unknown reasons and at (apparently) random times, Traefik fails to correctly and coimpletely write the certs to the `acme.json` file, resulting in an incomplete ans so invalid JSON file. 

If I `cat` the `acme.json` file, I can clearly see that the file is invalid, like so :

```jsonc
{
  "acmeResolver": {
    "Account": {
      "Email": "[REDACTED]",
      "Registration": {
        "body": {
          "status": "valid",
          "contact": [
            "mailto:[REDACTED]"
          ]
        },
        "uri": "https://acme-v02.api.letsencrypt.org/acme/acct/[REDACTED]"
      },
      "PrivateKey": "[REDACTED]",
      "KeyType": "4096"
    },
    "Certificates": [
      // For simplicity, only one shown here, but imagine ~100 elements in the array.
      {
        "domain": {
          "main": "[REDACTED]"
        },
        "certificate": "[REDACTED]",
        "key": "someKeyStringThatsRedacted....dISnJVMF
>>EOF

// OBVIOUSLY SOME STUFF IS MISSING HERE
// EOF is indicative, it doesn't appear when using `cat`
```
The cut certificate array element is the $`N`$th one in the array, where $`1 < N < 100`$

I have found no clue to why this occurs. Theories I've come up with up until now are :

- Concurrency issues if Traefik attempts two cert writes simultaneously, or any other concurrency issue
- Traefik crashes during the write process
- Traefik container restarts during the write process

I haven't looked over Traefik's code yet, but if the write process is streamed or chunked, a single sync write to the `acme.json` might fix this. Of course this might cause other issues.

### What version of Traefik are you using?

This issue has occurred over the years since I've updated Traefik from v1.7 to v2.5.
This issue has occurred on v2.5, v2.10 & v2.11.

### What is your environment & configuration?

```yml
log:
  level: ERROR
global:
  checkNewVersion: false
  sendAnonymousUsage: false
api:
  insecure: false
  dashboard: true
  debug: false
ping:
  entryPoint: traefik
providers:
  file:
    directory: /etc/traefik/dynamic
    watch: true
  docker:
    endpoint: unix:///var/run/docker.sock
    exposedByDefault: false
certificatesResolvers:
  acmeResolver:
    acme:
      email: [REDACTED]
      storage: /etc/traefik/acme.json
      httpChallenge:
        entryPoint: web
      caServer: https://acme-v02.api.letsencrypt.org/directory
defaultEntryPoints:
  - websecure
  - web
  - traefik
entryPoints:
  web:
    address: :80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true
  websecure:
    address: :443
    http:
      tls:
        certResolver: acmeResolver
  traefik:
    address: :8080
```

### If applicable, please paste the log output in DEBUG level

No really obvious log can be found relating to this issue except the "Can't read ACME file" once the issue has occured. 

My Traefik containers are configured to automatically restart on crashes (Docker's `restart: always` policy). Logs often get squashed by repeated crash logs. 


Discussion / hints:
Hi @Clovel,
Is the `acme.json` shared across multiple Traefik instances?
If yes, that's the reason of your issue:
https://doc.traefik.io/traefik/https/acme/#storage
> For concurrency reasons, this file cannot be shared across multiple instances of Traefik.


Hi @emilevauge,

No, the `acme.json` file is not shared across multiple Traefik instances. 

I do have another container reading that file on rare occasions to check if it has been broken or not to restore a backup of the file if necessary, but **_this bug existed before this feature in my app_**. This feature is a workaround to fall back on my feet if the `acme.json` file is corrupt. 
Can reading the file cause these concurrency issues ? To read it I use a basic `fs.promises.readFile` call in NodeJS v14 :

```ts
const lACMEFileContents: string = await fs.promises.readFile(
  TRAEFIK_ACME_CONFIG_FILE_PATH,
  {
    encoding: 'utf8',
    flag: 'r',
  },
);
```

I then backup the file as such just in case (mind that this will not be the backup I restore if it is broken) :
```ts
/* Write the ACME file to the backup directory */
await fs.promises.writeFile(
  `${TRAEFIK_ACME_CONFIG_FILE_PATH}.backup`,
  lACMEFileContents,
  {
    encoding: 'utf8',
    flag: 'w',
  },
);
```

Maybe the fact that it is in the same directoy might cause issues ?

Then I try to parse the file, and if it fails I then rewrite it :

```ts
let lParsedACMEFileContents: TraefikACME | null = null;
try {
  lParsedACMEFileContents = JSON.parse(lACMEFileContents) as TraefikACME;
} catch(pException) {
  console.error(`[ERROR] <TraefikManager.backupACMEFile> Failed to parse ACME file contents :`, pException);
  throw pException;
}
```

I only rewrite to the file if it is broken, using a previous backup that is valid. 
Hey @Clovel,

Thank you for your feedback.

To move forward on the analysis, we'd like to know if you have an idea of the best way to reproduce the issue and if you could provide a reproducible case (for instance, full Kubernetes manifest to reproduce the issue).

In the meantime, if any community member can help us find verified steps to reproduce and fix the issue if possible, we would love the help.

Thanks in advance.
> To move forward on the analysis, we'd like to know if you have an idea of the best way to reproduce the issue and if you could provide a reproducible case

To be honest, I don't have a reproduction process for this bug. For the moment, it appears to be random (of course it most likely isn't). I haven't found a specific case that causes this issue. 

If I find anything, I'll be sure to update you on this thread. 

> for instance, full Kubernetes manifest to reproduce the issue

I don't use K8s, but raw Docker on Linux (Debian 12). I deploy the various proxied apps via a (containerized) NodeJS app that uses the [Dockerode](https://github.com/apocas/dockerode) library. BTW, this app deploys Traefik when it's absent.

Here is an excerpts of the dynamic configuration (File Provider) : 

```yml
http:
  routers:
    FXw8Xfh7brjjkNo4p:
      rule: Host(`[REDACTED]`,`[REDACTED]`)
      service: FXw8Xfh7brjjkNo4p
      tls:
        certResolver: acmeResolver
        domains:
          - main: [REDACTED]
            sans:
              - [REDACTED]
      middlewares:
        - waf
...
...
  services:
    FXw8Xfh7brjjkNo4p:
      loadBalancer:
        servers:
          - url: http://172.18.0.11:80
...
  middlewares:
    waf:
      plugin:
        traefik-modsecurity-plugin:
          MaxBodySize: 10485760
          # And more redacted variables related to ModSecurity plugin
```

I'd like to point out that all the middleware stuff about ModSecurity was added later. The bug this thread is all about predates these features. 

**EDIT** : As stated earlier in the thread, each Traefik container manages about 100-150 containers per server. Each server has one (and only one) Traefik container. 
Hey @Clovel,

Thank you for your feedback.
We are waiting for more news from your side with a working example to test it.
I don't really know where to start to build a reproductible case here. 

What would you recommend ?

I almost need to have a full server of about 100 Docker containers to deploy, 1 or 2 domain names for each container to use for cert challenges, and an app to deploy all of these to have something that fits my case. The test case is my app so to speak...

We'd have to mock almost all the components to have a solid test case, which would require a lot of time. I'm sure there is a better way to build a test case.

Or maybe by analyzing the code that manages the writes to the ACME file we can find something worthy of an upgrade/fix. 
Hello @Clovel,

> What would you recommend ?

TBH this issue is hard to reproduce. We haven't been able to set up a reproducible use case.
For this reason, if you face the issue again, we'd be interested in having more information on the environment to move forward.

Please keep us updated even if you do not face the problem again (in such a case we'll close the issue).
In the meantime, I left the issue open, waiting for more news from you.
> Please keep us updated even if you do not face the problem again (in such a case we'll close the issue).

Well, I had it twice last week on two different servers, but no logs or specific events seem to point to a cause. 

I will keep you updated. 

P.S. : If I ever have the time, I'll try too look at the Traefik source code too, Maybe something is obvious (not necessarily a bug, but an explanation for this behaviour)
Still haven't found anything. 

I was wondering : Does the Traefik source code ensure atomic writing into the `acme.json` file ? In other words, if Traefik starts to write to the file and shuts down (`docker restart` for example), does it gracefully wait for the end of the write operation or does it interrupt the write operation ?
Hi! I'm Træfiker :robot: the bot in charge of tidying up the issues.I have to close this one because of its lack of activity :disappointed:Feel free to re-open it or join our [Community Forum](https://community.containo.us/).  

Fix patch:
diff --git a/cmd/traefik/traefik.go b/cmd/traefik/traefik.go
index d25951b933..01dbb9ed21 100644
--- a/cmd/traefik/traefik.go
+++ b/cmd/traefik/traefik.go
@@ -188,7 +188,7 @@ func setupServer(staticConfiguration *static.Configuration) (*server.Server, err
 		return nil, err
 	}
 
-	acmeProviders := initACMEProvider(staticConfiguration, providerAggregator, tlsManager, httpChallengeProvider, tlsChallengeProvider)
+	acmeProviders := initACMEProvider(staticConfiguration, providerAggregator, tlsManager, httpChallengeProvider, tlsChallengeProvider, routinesPool)
 
 	// Entrypoints
 
@@ -366,7 +366,7 @@ func switchRouter(routerFactory *server.RouterFactory, serverEntryPointsTCP serv
 }
 
 // initACMEProvider creates an acme provider from the ACME part of globalConfiguration.
-func initACMEProvider(c *static.Configuration, providerAggregator *aggregator.ProviderAggregator, tlsManager *traefiktls.Manager, httpChallengeProvider, tlsChallengeProvider challenge.Provider) []*acme.Provider {
+func initACMEProvider(c *static.Configuration, providerAggregator *aggregator.ProviderAggregator, tlsManager *traefiktls.Manager, httpChallengeProvider, tlsChallengeProvider challenge.Provider, routinesPool *safe.Pool) []*acme.Provider {
 	localStores := map[string]*acme.LocalStore{}
 
 	var resolvers []*acme.Provider
@@ -376,7 +376,7 @@ func initACMEProvider(c *static.Configuration, providerAggregator *aggregator.Pr
 		}
 
 		if localStores[resolver.ACME.Storage] == nil {
-			localStores[resolver.ACME.Storage] = acme.NewLocalStore(resolver.ACME.Storage)
+			localStores[resolver.ACME.Storage] = acme.NewLocalStore(resolver.ACME.Storage, routinesPool)
 		}
 
 		p := &acme.Provider{
diff --git a/pkg/provider/acme/local_store.go b/pkg/provider/acme/local_store.go
index 33084b3204..9a75c704ee 100644
--- a/pkg/provider/acme/local_store.go
+++ b/pkg/provider/acme/local_store.go
@@ -1,6 +1,7 @@
 package acme
 
 import (
+	"context"
 	"encoding/json"
 	"io"
 	"os"
@@ -22,9 +23,9 @@ type LocalStore struct {
 }
 
 // NewLocalStore initializes a new LocalStore with a file name.
-func NewLocalStore(filename string) *LocalStore {
+func NewLocalStore(filename string, routinesPool *safe.Pool) *LocalStore {
 	store := &LocalStore{filename: filename, saveDataChan: make(chan map[string]*StoredData)}
-	store.listenSaveAction()
+	store.listenSaveAction(routinesPool)
 	return store
 }
 
@@ -99,18 +100,31 @@ func (s *LocalStore) get(resolverName string) (*StoredData, error) {
 }
 
 // listenSaveAction listens to a chan to store ACME data in json format into `LocalStore.filename`.
-func (s *LocalStore) listenSaveAction() {
-	safe.Go(func() {
+func (s *LocalStore) listenSaveAction(routinesPool *safe.Pool) {
+	routinesPool.GoCtx(func(ctx context.Context) {
 		logger := log.WithoutContext().WithField(log.ProviderName, "acme")
-		for object := range s.saveDataChan {
-			data, err := json.MarshalIndent(object, "", "  ")
-			if err != nil {
-				logger.Error(err)
-			}
+		for {
+			select {
+			case <-ctx.Done():
+				return
+
+			case object := <-s.saveDataChan:
+				select {
+				case <-ctx.Done():
+					// Stop handling events because Traefik is shutting down.
+					return
+				default:
+				}
 
-			err = os.WriteFile(s.filename, data, 0o600)
-			if err != nil {
-				logger.Error(err)
+				data, err := json.MarshalIndent(object, "", "  ")
+				if err != nil {
+					logger.Error(err)
+				}
+
+				err = os.WriteFile(s.filename, data, 0o600)
+				if err != nil {
+					logger.Error(err)
+				}
 			}
 		}
 	})


Test patch:
diff --git a/pkg/provider/acme/local_store_test.go b/pkg/provider/acme/local_store_test.go
index 4614078c0a..61d2cff6bd 100644
--- a/pkg/provider/acme/local_store_test.go
+++ b/pkg/provider/acme/local_store_test.go
@@ -1,6 +1,7 @@
 package acme
 
 import (
+	"context"
 	"fmt"
 	"os"
 	"path/filepath"
@@ -9,6 +10,7 @@ import (
 
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+	"github.com/traefik/traefik/v2/pkg/safe"
 )
 
 func TestLocalStore_GetAccount(t *testing.T) {
@@ -45,7 +47,7 @@ func TestLocalStore_GetAccount(t *testing.T) {
 
 	for _, test := range testCases {
 		t.Run(test.desc, func(t *testing.T) {
-			s := NewLocalStore(test.filename)
+			s := NewLocalStore(test.filename, safe.NewPool(context.Background()))
 
 			account, err := s.GetAccount("test")
 			require.NoError(t, err)
@@ -58,7 +60,7 @@ func TestLocalStore_GetAccount(t *testing.T) {
 func TestLocalStore_SaveAccount(t *testing.T) {
 	acmeFile := filepath.Join(t.TempDir(), "acme.json")
 
-	s := NewLocalStore(acmeFile)
+	s := NewLocalStore(acmeFile, safe.NewPool(context.Background()))
 
 	email := "some@email.com"