Repository: wagoodman/dive (language: go)

Issue:
Unable to analyze the image using dive
**What happened**:  I am using RHEL9  free tier from AWS and installed the dive with below steps by referring the github page https://github.com/wagoodman/dive  and the version is dive 0.11.0
i have downloaded the Docker image "nginx:latest" by using "docker pull nginx" command
then i have executed "dive nginx:latest", getting below error

[ec2-user@ip-172-31-36-53 ~]$ sudo dive nginx
Image Source: docker://nginx
Fetching image... (this can take a while for large images)
cannot fetch image
could not find image config
[ec2-user@ip-172-31-36-53 ~]$

Could you please let me know how to scan an image using dive?



**Anything else we need to know?**: I am interested to know how to include the dive into jenkins CI/CD pipeline.  like we are using Free style jobs so need to include the configuration in the "execute shell"

**Environment**: RHEL9
- OS version: 
              - [ec2-user@ip-172-31-36-53 ~]$ uname -a
              Linux ip-172-31-36-53.ec2.internal 5.14.0-362.8.1.el9_3.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Oct 3 11:12:36 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux
              [ec2-user@ip-172-31-36-53 ~]$


- Docker version (if applicable)
              - [ec2-user@ip-172-31-36-53 ~]$ docker --version
              Docker version 25.0.0, build e758fe5
[ec2-user@ip-172-31-36-53 ~]$




Discussion / hints:
Could it be that this is related to a Docker update that was released last weekend? I also see the same issue on Docker v25.0.0 but haven't yet seen in on v24.0.7.
@rajiv-k is this something you could look into?
Apparently, the files no longer follow the same naming logic, so [this code](https://github.com/wagoodman/dive/blob/3ef1dd2c74c6952064d866adc2b225048d939f83/dive/image/docker/image_archive.go#L79-L85) no longer detects the config file, and [this one](https://github.com/wagoodman/dive/blob/3ef1dd2c74c6952064d866adc2b225048d939f83/dive/image/docker/image_archive.go#L96-L99) later fails.

Here's the same image (same Dockerfile) built with Docker 24.0.7 (pulled from our registry last week):
```
$ tar tf redacted.tar 
41b7e5a676314cb0e6b37013986b3abc3175bd1711e54574baf2619b56e00d4b/
41b7e5a676314cb0e6b37013986b3abc3175bd1711e54574baf2619b56e00d4b/VERSION
41b7e5a676314cb0e6b37013986b3abc3175bd1711e54574baf2619b56e00d4b/json
41b7e5a676314cb0e6b37013986b3abc3175bd1711e54574baf2619b56e00d4b/layer.tar
8d179503ce3761ab615a6c26c87cb707d9e622b5862ee09e3ee020895b06d545.json
e2745f779510e653aa2510f80e46e95a450d990ba58d40439e2a173dd389dbc8/
e2745f779510e653aa2510f80e46e95a450d990ba58d40439e2a173dd389dbc8/VERSION
e2745f779510e653aa2510f80e46e95a450d990ba58d40439e2a173dd389dbc8/json
e2745f779510e653aa2510f80e46e95a450d990ba58d40439e2a173dd389dbc8/layer.tar
ec777bf3518453637ef782f8888a75ac522f0c6aaf033ee3a26214d1f6ce463f/
ec777bf3518453637ef782f8888a75ac522f0c6aaf033ee3a26214d1f6ce463f/VERSION
ec777bf3518453637ef782f8888a75ac522f0c6aaf033ee3a26214d1f6ce463f/json
ec777bf3518453637ef782f8888a75ac522f0c6aaf033ee3a26214d1f6ce463f/layer.tar
manifest.json
repositories
```
and saved from a failed CI build today, with Docker 25.0.0:
```
$ tar tf xxx.tar 
blobs/
blobs/sha256/
blobs/sha256/07f6577589fb824de3bf45f2f7954fcf108bae5a6fc3b44e16029729733c454b
blobs/sha256/4dce6976862b95892955b4bd419eeff40d30fbe21fd1bbcefde6bf8b3e9491e0
blobs/sha256/583c2b89a34971925363d1f3de987bb19c907b98f47b808beb8954f699943913
blobs/sha256/8e87ff28f1b5ff2d5131999ccfa1e674cb252631c50683f5ee43fad59cbea8e1
blobs/sha256/9c7350069c7fb5363a35bbf8217870ee7d0f6cc839d330c64a4d13a306edcd8c
blobs/sha256/a08f2f0d2d5e0ed1f1b74bb62a504ff68cf3ea7f803060a0ca1340109bbc99d0
blobs/sha256/cc52537ba04fe1c94df2117aa105dfa740287df3069206ce6c95873dd5b7d350
blobs/sha256/db7eb0536bb70e202a19fef88169b53b656eeba01899f3f4dcb8f5dd7191e6ce
index.json
manifest.json
oci-layout
repositories
```
In the first case, the config file is that `8d179503ce3761ab615a6c26c87cb707d9e622b5862ee09e3ee020895b06d545.json` at the root of the archive, in the second, it's the `blobs/sha256/a08f2f0d2d5e0ed1f1b74bb62a504ff68cf3ea7f803060a0ca1340109bbc99d0` file. Note how we can no longer tell just from their name which files are JSON and which are TAR files.

This is I believe because:
> The `docker image save` tarball output is now OCI compliant. [moby/moby#44598](https://github.com/moby/moby/pull/44598)

(https://docs.docker.com/engine/release-notes/25.0/)
Same here using dive 0.11.0 as docker image, Docker version 25.0.0, build e758fe5, Ubuntu
@tbroyer since you've already _dived_ into the relevant code, are you interested in sending a PR?
@mark2185 I started looking at it yesterday, but am really not sure how to approach it: start reading each file in the tar looking for a "magic number"? (knowing that tar and json don't really have such things) and/or trying to load them as tar or json and ignoring errors? (this could imply buffering a big chunk of the file) should this new/fallback approach be limited to entries in `blobs/`? Also, I haven't written Go in years.

I'll happily review a PR and test it though, and could help brainstorm the best approach.
Fair enough, wanted to be sure before taking a crack at it.

I'd wager a good way would be initializing a `tar.NewReader` with the buffer and invoking `Next()`, if there's an error (that's not a `io.EOF`) we're definitely _not_ reading a `tar` and hopefully we're reading a JSON.

Then we try unmarshalling it to see how well that works out, and if that errors out as well we can safely say _what the hell_.
You have to take gzip into account too, and remember that in many (most?) cases we're reading a tar that's directly streamed from the Docker daemon, and is not seekable. This means adding some buffering; 512 bytes should be enough as that's the size of a tar entry header (according to https://en.wikipedia.org/wiki/Tar_(computing)#File_format), but what if it's gzipped? maybe in this case it can be assumed to **not** be JSON so the fallback to JSON, and need for seeking back to the start of the entry, is not needed?

That would mean doing something like:

```
for each tar entry:
  // use the current algorithm based on file names, then do the following:
  else: // handle OCI-compatible Docker images
    buffer = read 512 bytes
    if buffer matches gzip magic number:
      create gzip stream (from a MultiReader on the buffer and remaining of the tar entry)
      create tar reader
      try processing layer, continue to next tar entry on error (ignoring it)
    else:
      create a tar reader (from a MultiReader on the buffer and remaining of the tar entry)
      try processing layer
      on error, try processing JSON (from a MultiReader on the buffer and remaining of the tar entry; assuming/hoping the previous tar reader didn't consume past the buffer)
```
@tbroyer if I'm reading the [image layout specification](https://github.com/opencontainers/image-spec/blob/970322241ad9c8868d15e6859c9b446ad06b944d/image-layout.md#indexjson-file) correctly, the `index.json` should have the `mediaType` indicating which type it is. 

Can't check because I don't have such an image, which leads me to my next question - where did you get `docker v25`? 
I have `v24.0.7` on `archlinux` and a couple of friends I asked (windows, macOS) don't have `v25` either.
> @tbroyer if I'm reading the [image layout specification](https://github.com/opencontainers/image-spec/blob/970322241ad9c8868d15e6859c9b446ad06b944d/image-layout.md#indexjson-file) correctly, the `index.json` should have the `mediaType` indicating which type it is.

Yes, **but** nothing guarantees that you'll see the `index.json` tar entry before the others, so either you possibly process the tar twice (once to get the index –you could also get the `manifest.json` instead– and then again looking up each tar entry in the index/manifest to know how to process it …but that means storing the tar to disk temporarily when resolving it from the Docker daemon), or you "content sniff" and parse/process each tar entry (like the pseudo algorithm above)
Maybe extracting the index or manifest first would be the best way forward, but that's a major change.

> Can't check because I don't have such an image, which leads me to my next question - where did you get `docker v25`? I have `v24.0.7` on `archlinux` and a couple of friends I asked (windows, macOS) don't have `v25` either.

I too am on Arch so still on 24.0.7, our CI is on Ubuntu though, and we get the Docker Engine packages straight from Docker, Inc.: https://docs.docker.com/engine/install/ubuntu/
I managed to generate the `.data/test-docker-image.tar` on our CI server using `make generate-test-data` (I created an empty `.scripts/` so it's not exactly equivalent to the TAR that's checked into the repository)
Note that I had to ZIP the TAR for GitHub to accept the upload :man_shrugging: 
[test-docker-image.zip](https://github.com/wagoodman/dive/files/14024321/test-docker-image.zip)

And I reproduce the issue with it:
```
$ dive --source docker-archive test-docker-image.tar 
Image Source: docker-archive://test-docker-image.tar
Fetching image... (this can take a while for large images)
cannot fetch image
could not find image config
```
I'm on ubuntu 22.04 using docker `25.0.0` and run into this problem. Can I help in any way?
I have same problem 

```
$ dive ubuntu:22.04
Image Source: docker://ubuntu:22.04
Fetching image... (this can take a while for large images)
Handler not available locally. Trying to pull 'ubuntu:22.04'...
22.04: Pulling from library/ubuntu
29202e855b20: Pull complete 
Digest: sha256:e6173d4dc55e76b87c4af8db8821b1feae4146dd47341e4d431118c7dd060a74
Status: Downloaded newer image for ubuntu:22.04
docker.io/library/ubuntu:22.04
cannot fetch image
could not find image config
```
System info:

```
Debian GNU/Linux 12
Docker version 25.0.1, build 29cf629
dive 0.11.0
```

Fix patch:
diff --git a/dive/image/docker/image_archive.go b/dive/image/docker/image_archive.go
index 3a818737..827b58ba 100644
--- a/dive/image/docker/image_archive.go
+++ b/dive/image/docker/image_archive.go
@@ -2,7 +2,9 @@ package docker
 
 import (
 	"archive/tar"
+	"bytes"
 	"compress/gzip"
+	"encoding/json"
 	"fmt"
 	"io"
 	"os"
@@ -46,6 +48,7 @@ func NewImageArchive(tarFile io.ReadCloser) (*ImageArchive, error) {
 
 		// some layer tars can be relative layer symlinks to other layer tars
 		if header.Typeflag == tar.TypeSymlink || header.Typeflag == tar.TypeReg {
+			// For the Docker image format, use file name conventions
 			if strings.HasSuffix(name, ".tar") {
 				currentLayer++
 				layerReader := tar.NewReader(tarReader)
@@ -82,6 +85,55 @@ func NewImageArchive(tarFile io.ReadCloser) (*ImageArchive, error) {
 					return img, err
 				}
 				jsonFiles[name] = fileBuffer
+			} else if strings.HasPrefix(name, "blobs/") {
+				// For the OCI-compatible image format (used since Docker 25), use mime sniffing
+				// but limit this to only the blobs/ (containing the config, and the layers)
+
+				// The idea here is that we try various formats in turn, and those tries should
+				// never consume more bytes than this buffer contains so we can start again.
+
+				// 512 bytes ought to be enough (as that's the size of a TAR entry header),
+				// but play it safe with 1024 bytes. This should also include very small layers
+				// (unless they've also been gzipped, but Docker does not appear to do it)
+				buffer := make([]byte, 1024)
+				n, err := io.ReadFull(tarReader, buffer)
+				if err != nil && err != io.ErrUnexpectedEOF {
+					return img, err
+				}
+
+				// Only try reading a TAR if file is "big enough"
+				if n == cap(buffer) {
+					var unwrappedReader io.Reader
+					unwrappedReader, err = gzip.NewReader(io.MultiReader(bytes.NewReader(buffer[:n]), tarReader))
+					if err != nil {
+						// Not a gzipped entry
+						unwrappedReader = io.MultiReader(bytes.NewReader(buffer[:n]), tarReader)
+					}
+
+					// Try reading a TAR
+					layerReader := tar.NewReader(unwrappedReader)
+					tree, err := processLayerTar(name, layerReader)
+					if err == nil {
+						currentLayer++
+						// add the layer to the image
+						img.layerMap[tree.Name] = tree
+						continue
+					}
+				}
+
+				// Not a TAR (or smaller than our buffer), might be a JSON file
+				decoder := json.NewDecoder(bytes.NewReader(buffer[:n]))
+				token, err := decoder.Token()
+				if _, ok := token.(json.Delim); err == nil && ok {
+					// Looks like a JSON object (or array)
+					// XXX: should we add a header.Size check too?
+					fileBuffer, err := io.ReadAll(io.MultiReader(bytes.NewReader(buffer[:n]), tarReader))
+					if err != nil {
+						return img, err
+					}
+					jsonFiles[name] = fileBuffer
+				}
+				// Ignore every other unknown file type
 			}
 		}
 	}


Test patch:
diff --git a/.data/test-oci-docker-image.tar b/.data/test-oci-docker-image.tar
new file mode 100644
index 00000000..d8ea90de
Binary files /dev/null and b/.data/test-oci-docker-image.tar differ