Repository: moby/moby (language: go)

Issue:
build history: orphan records when switching between graphdriver and containerd store
### Description

relates to
* https://github.com/docker/for-mac/issues/7400
* https://github.com/docker/for-win/issues/14288

When switching to another store (graphdriver or containerd), builds from past store cannot be accessed / removed anymore and vice-versa.

### Reproduce

Start a dind container with graphdriver store:

```console
docker run -d --restart always --privileged --name docker27 -v docker27dt:/var/lib/docker -p 12375:2375 docker:27-dind --debug --host=tcp://0.0.0.0:2375 --tlsverify=false
```

Create a docker context and switch to it so buildx can build on it:

```console
docker context create docker27 --docker "host=tcp://localhost:12375,skip-tls-verify=true"
docker context use docker27
```

Do a build with this Dockerfile:

```Dockerfile
FROM busybox
RUN echo "Hello, World!"
```

```console
docker buildx build .
```

You should see the build on Docker Desktop and open it:

![450e58d1-d678-4a95-b8db-ab8d1010894a](https://github.com/user-attachments/assets/ae6dc2a3-787d-4e65-a4b4-d404f9ffe4bb)

![53f6071f-a614-4b92-8204-ed4a1bf864e6](https://github.com/user-attachments/assets/37c03ef7-a796-4d69-9086-7c699e054199)

Now restart the container with containerd store enabled:

```console
docker context use default
docker rm -f docker27
docker run -d --restart always --privileged --name docker27 -v docker27dt:/var/lib/docker -p 12375:2375 --env TEST_INTEGRATION_USE_SNAPSHOTTER=1 docker:27-dind --debug --host=tcp://0.0.0.0:2375 --tlsverify=false
```

You should see containerd store enabled in the logs:

```
time="2024-09-30T19:43:01.560448969Z" level=info msg="Starting daemon with containerd snapshotter integration enabled"
```

We still see the build in Docker Dekstop but opening it yield to this error:

![831e71d2-fc32-461f-9bd4-270469f67df7](https://github.com/user-attachments/assets/663f3a59-e3df-4041-b44c-a407d9274070)




### Expected behavior

As the blobs for this record are in another store I would expect it not being listed by `ControlClient().ListenBuildHistory(ctx, &controlapi.BuildHistoryRequest{})`.

### docker version

```bash
Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:02 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```


### docker info

```bash
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.3.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 6.10.9-linuxkit
 Operating System: Alpine Linux v3.20 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 31.3GiB
 Name: 0d137b8046d0
 ID: af60703c-6a88-4bf5-8dd7-53880e95d58b
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 34
  Goroutines: 69
  System Time: 2024-09-30T19:54:49.198771943Z
  EventsListeners: 0
 Username: crazymax
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
```


### Additional Info

I think we have several ways to mitigate this issue:
* Even if the record is listed in the history db, check if it is part of the current content store before sending the event.
* Create a new "store" field in the history db bucket and do a migration of all records that would set the db being used (either "graph" or "containerd") for each of them.
  * Or create a bbolt history db for each store and do the migration?

cc @tonistiigi 


Discussion / hints:
The issue is that graphdriver and containerd storage modes use different contentstore. So the blobs created there, like the progress stream or otel trace, go to different locations. But the boltdb for `history.db` is started at the same path for both.

My suggestion is to make it so that the path is different for containerd storage, eg. `history_ctd.db`. This means that when switching stores, the old history items would disappear and leak storage, but that doesn't look any different from the other grphdriver/image data atm.

https://github.com/moby/moby/blob/v27.3.1/builder/builder-next/controller.go#L201
> My suggestion is to make it so that the path is different for containerd storage, eg. `history_ctd.db`. This means that when switching stores, the old history items would disappear and leak storage, but that doesn't look any different from the other grphdriver/image data atm.

Sounds good but should we do a migration? Meaning move entries from `history.db` that don't belong to graph driver to `history_ctd.db`? Would need to open both content store though to check that or maybe just open containerd content store and move history db entries matching digest?
If there is any migration, it would need to be between the contentstores/snapshotters . But as long as this does not happen for any other data, don't see a reason to have it for history.
> don't see a reason to have it for history.

It's just that `ListenBuildHistory` will always return records that may contain inaccessible blobs. This could be confusing for users that will check history with https://github.com/docker/buildx/issues/2711
If you move `history.db` to another URL then `ListenBuildHistory` will not return them anymore. `ListenBuildHistory` reads records from the boltdb.
> If you move `history.db` to another URL then `ListenBuildHistory` will not return them anymore. `ListenBuildHistory` reads records from the boltdb.

Yes but some records from current `history.db` could still be erroneous iiuc

Fix patch:
diff --git a/builder/builder-next/controller.go b/builder/builder-next/controller.go
index 4c38642d9d22c..0dea764b7d3f3 100644
--- a/builder/builder-next/controller.go
+++ b/builder/builder-next/controller.go
@@ -43,6 +43,8 @@ import (
 	containerdsnapshot "github.com/moby/buildkit/snapshot/containerd"
 	"github.com/moby/buildkit/solver"
 	"github.com/moby/buildkit/solver/bboltcachestorage"
+	"github.com/moby/buildkit/solver/pb"
+	"github.com/moby/buildkit/util/apicaps"
 	"github.com/moby/buildkit/util/archutil"
 	"github.com/moby/buildkit/util/entitlements"
 	"github.com/moby/buildkit/util/network/netproviders"
@@ -55,9 +57,6 @@ import (
 	"go.etcd.io/bbolt"
 	bolt "go.etcd.io/bbolt"
 	"go.opentelemetry.io/otel/sdk/trace"
-
-	"github.com/moby/buildkit/solver/pb"
-	"github.com/moby/buildkit/util/apicaps"
 )
 
 func newController(ctx context.Context, rt http.RoundTripper, opt Opt) (*control.Controller, error) {
@@ -86,7 +85,7 @@ func newSnapshotterController(ctx context.Context, rt http.RoundTripper, opt Opt
 		return nil, err
 	}
 
-	historyDB, historyConf, err := openHistoryDB(opt.Root, opt.BuilderConfig.History)
+	historyDB, historyConf, err := openHistoryDB(opt.Root, "history_c8d.db", opt.BuilderConfig.History)
 	if err != nil {
 		return nil, err
 	}
@@ -197,8 +196,8 @@ func newSnapshotterController(ctx context.Context, rt http.RoundTripper, opt Opt
 	})
 }
 
-func openHistoryDB(root string, cfg *config.BuilderHistoryConfig) (*bolt.DB, *bkconfig.HistoryConfig, error) {
-	db, err := bbolt.Open(filepath.Join(root, "history.db"), 0o600, nil)
+func openHistoryDB(root string, fn string, cfg *config.BuilderHistoryConfig) (*bolt.DB, *bkconfig.HistoryConfig, error) {
+	db, err := bbolt.Open(filepath.Join(root, fn), 0o600, nil)
 	if err != nil {
 		return nil, nil, err
 	}
@@ -344,7 +343,7 @@ func newGraphDriverController(ctx context.Context, rt http.RoundTripper, opt Opt
 		return nil, err
 	}
 
-	historyDB, historyConf, err := openHistoryDB(opt.Root, opt.BuilderConfig.History)
+	historyDB, historyConf, err := openHistoryDB(opt.Root, "history.db", opt.BuilderConfig.History)
 	if err != nil {
 		return nil, err
 	}


Test patch: