Repository: google/osv-scanner

Issue:
Support snapshot artifacts when fetching Maven parent POMs
Hello,

I encountered an issue while performing a security scan on one of my Java projects using `osv-scanner`. The scan failed, and the error output is as follows:

```
❯ ./osv-scanner -skip-git -r $HOME/gitlab/abcd/broker/my-kyc-center
Scanning dir /Users/user/gitlab/abcd/broker/my-kyc-center
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-api/pom.xml
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-common/pom.xml
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-job/pom.xml
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-persist/pom.xml
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-service/pom.xml
Scanning directory for vendored libs: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-service/src/main/java/com/my/kyc/service/vendor
Scanning potential vendored dir: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-service/src/main/java/com/my/kyc/service/vendor/model
Scanning potential vendored dir: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-service/src/main/java/com/my/kyc/service/vendor/security
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-web/pom.xml
Attempted to scan lockfile but failed: /Users/user/gitlab/abcd/broker/my-kyc-center/pom.xml
No package sources found, --help for usage information.
```

After debugging the source code, I obtained more detailed error information:

```
"failed extracting /Users/user/gitlab/abcd/broker/my-kyc-center/kyc-api/pom.xml: failed to merge parents: failed to get Maven project com.my.kyc:kyc-center:2.0.0-SNAPSHOT: API query failed: Maven registry query status: 404 Not Found"
```

The directory `/Users/user/gitlab/abcd/broker/my-kyc-center` contains a multi-module Maven project. Through debugging, I think I discovered the following two points causing the scan failure:

1. **Hardcoded Maven Central URL:**
   In the file [maven_registry.go](https://github.com/google/osv-scanner/blob/main/internal/resolution/datasource/maven_registry.go#L15), the Maven Central URL is hardcoded. Since my project is a private project and only submitted to the internal Nexus repository, directly querying the Maven Central results in a query failure and an error. I modified `const MavenCentral` to the internal Nexus URL and continued running, but found the same error, which led to discovering the second issue.

2. **Snapshot Version Handling:**
   When requesting the Maven repository for snapshot versions, the `maven-metadata.xml` file should be requested first. For example, the current `GetProject` function in the `maven_registry.go` file directly requests:
   ```
   https://nexus/repository/abcd-prod-proxy/com/my/kyc/kyc-center/2.0.0-SNAPSHOT/kyc-center-2.0.0-SNAPSHOT.pom
   ```
   This results in a 404 not found error because if there are multiple submissions of `2.0.0-SNAPSHOT`, the POM file cannot be directly obtained. Instead, the `maven-metadata.xml` file at:
   ```
   https://nexus/repository/abcd-prod-proxy/com/my/kyc/kyc-center/2.0.0-SNAPSHOT/maven-metadata.xml
   ```
   should be read for the `snapshotVersion`, for example:
   ```xml
   <snapshotVersions>
     <snapshotVersion>
       <extension>pom</extension>
       <value>2.0.0-20240711.060746-8</value>
       <updated>20240711060746</updated>
     </snapshotVersion>
   </snapshotVersions>
   ```
   Then, the correct POM file can be requested:
   ```
   https://nexus/repository/abcd-prod-proxy/com/my/kyc/kyc-center/2.0.0-SNAPSHOT/kyc-center-2.0.0-20240711.060746-8.pom
   ```
### Environment Information:
Operating System: macOS 14.5
Java Version: openjdk version "11.0.21"
Maven Version: Apache Maven 3.9.6 

Please let me know if you need any additional information or if there are any other steps I can take to assist in resolving this issue.

Thank you for your attention to this matter!

Hints from issue comments:
@another-rex: Thanks for the detailed report! Can you also add the osv-scanner version you were using as well?
@cuixq can you take a look?@cuixq: I think this is related to this feature request - [supporting private registry for Maven](https://github.com/google/osv-scanner/issues/1045).

For the snapshot versions in `mavne-metadata.xml`, I think this is for snapshot remote repositories based on the [reference](https://maven.apache.org/repositories/metadata.html#the-v-level-metadata).  I will rename this bug to reflect supporting [consuming snapshot artifacts](https://maven.apache.org/repositories/remote.html#consuming-snapshot-artifacts).@cuixq: @Malayke do you know any documentation or more examples about this? The [link](https://repository.apache.org/content/repositories/snapshots/org/apache/maven/plugins/maven-jar-plugin/3.3.0-SNAPSHOT/maven-metadata.xml) on this [page](https://maven.apache.org/repositories/metadata.html#the-v-level-metadata) returns 404 :(@Malayke: Hi @cuixq , you can also access [this link](https://repository.apache.org/service/local/repositories/snapshots/content/org/apache/maven/plugins/maven-jar-plugin/3.3.1-SNAPSHOT/maven-metadata.xml), you find the right document, I discovered this mechanism independently by frequently running `mvn dependency:tree`. If the version includes `snapshot`, Maven first accesses `maven-metadata.xml`. Otherwise, it directly accesses the POM file.
You can find an explanation of this feature [here](https://strongbox.github.io/developer-guide/metadata/maven-metadata.html).@Malayke: Hi @cuixq  I was wondering if I might kindly inquire about the possibility of supporting a custom Maven Central URL?@cuixq: Hi @Malayke do you mean a custom Maven registry URL instead of the hard-coded Maven Central URL? We definitely would like to support that. @Malayke: yes, thank you for your previous response! I appreciate your work on supporting snapshot version fetching from Maven registries. It's a great feature!

I was wondering if it might be possible to consider adding support for custom Maven registry URLs, rather than using a hard-coded Maven Central URL? I noticed this wasn't mentioned in the current implementation, which is why I thought to ask about it.

I remember bringing this up in this issue, and I'm curious if there have been any thoughts or plans regarding this feature. It would be incredibly helpful for those of us who might need to use alternative Maven repositories.

I completely understand if this isn't currently on the roadmap, but I'd be grateful for any insights you could share on this topic. Thank you so much for your time and for considering my question!
@cuixq: Yes - supporting Maven private registries https://github.com/google/osv-scanner/issues/1045 is one of our priorities. We do plan to make the Maven repository URL configurable. I will update the mentioned bug with more details. Stay tuned! @tushar-panto: Does it  support maven private registries now? like did any upgrades happened @cuixq: You should be able to use [`--maven-registry`](https://google.github.io/osv-scanner/supported-languages-and-lockfiles/#data-source) to specify the URL of the private registry that you want to fetch Maven artifacts.

Solution patch:
diff --git a/internal/resolution/datasource/maven_registry.go b/internal/resolution/datasource/maven_registry.go
index bcc6309b..24c54d02 100644
--- a/internal/resolution/datasource/maven_registry.go
+++ b/internal/resolution/datasource/maven_registry.go
@@ -24,31 +24,97 @@ func NewMavenRegistryAPIClient(registry string) *MavenRegistryAPIClient {
 
 var errAPIFailed = errors.New("API query failed")
 
+// GetProject fetches a pom.xml specified by groupID, artifactID and version and parses it to maven.Project.
+// For a snapshot version, version level metadata is used to find the extact version string.
+// More about Maven Repository Metadata Model: https://maven.apache.org/ref/3.9.9/maven-repository-metadata/
+// More about Maven Metadata: https://maven.apache.org/repositories/metadata.html
 func (m *MavenRegistryAPIClient) GetProject(ctx context.Context, groupID, artifactID, version string) (maven.Project, error) {
-	u, err := url.JoinPath(m.registry, strings.ReplaceAll(groupID, ".", "/"), artifactID, version, fmt.Sprintf("%s-%s.pom", artifactID, version))
+	if !strings.HasSuffix(version, "-SNAPSHOT") {
+		return m.getProject(ctx, groupID, artifactID, version, "")
+	}
+
+	// Fetch version metadata for snapshot versions.
+	metadata, err := m.getVersionMetadata(ctx, groupID, artifactID, version)
+	if err != nil {
+		return maven.Project{}, err
+	}
+
+	snapshot := ""
+	for _, sv := range metadata.Versioning.SnapshotVersions {
+		if sv.Extension == "pom" {
+			// We only look for pom.xml for project metadata.
+			snapshot = string(sv.Value)
+			break
+		}
+	}
+
+	return m.getProject(ctx, groupID, artifactID, version, snapshot)
+}
+
+// getProject fetches a pom.xml specified by groupID, artifactID and version and parses it to maven.Project.
+// For snapshot versions, the exact version value is specified by snapshot.
+func (m *MavenRegistryAPIClient) getProject(ctx context.Context, groupID, artifactID, version, snapshot string) (maven.Project, error) {
+	if snapshot == "" {
+		snapshot = version
+	}
+	u, err := url.JoinPath(m.registry, strings.ReplaceAll(groupID, ".", "/"), artifactID, version, fmt.Sprintf("%s-%s.pom", artifactID, snapshot))
 	if err != nil {
 		return maven.Project{}, fmt.Errorf("failed to join path: %w", err)
 	}
 
-	req, err := http.NewRequestWithContext(ctx, http.MethodGet, u, nil)
+	var proj maven.Project
+	if err := get(ctx, u, &proj); err != nil {
+		return maven.Project{}, err
+	}
+
+	return proj, nil
+}
+
+// getVersionMetadata fetches a version level maven-metadata.xml and parses it to maven.Metadata.
+func (m *MavenRegistryAPIClient) getVersionMetadata(ctx context.Context, groupID, artifactID, version string) (maven.Metadata, error) {
+	u, err := url.JoinPath(m.registry, strings.ReplaceAll(groupID, ".", "/"), artifactID, version, "maven-metadata.xml")
+	if err != nil {
+		return maven.Metadata{}, fmt.Errorf("failed to join path: %w", err)
+	}
+
+	var metadata maven.Metadata
+	if err := get(ctx, u, &metadata); err != nil {
+		return maven.Metadata{}, err
+	}
+
+	return metadata, nil
+}
+
+// GetArtifactMetadata fetches an artifact level maven-metadata.xml and parses it to maven.Metadata.
+func (m *MavenRegistryAPIClient) GetArtifactMetadata(ctx context.Context, groupID, artifactID string) (maven.Metadata, error) {
+	u, err := url.JoinPath(m.registry, strings.ReplaceAll(groupID, ".", "/"), artifactID, "maven-metadata.xml")
 	if err != nil {
-		return maven.Project{}, fmt.Errorf("failed to make new request: %w", err)
+		return maven.Metadata{}, fmt.Errorf("failed to join path: %w", err)
+	}
+
+	var metadata maven.Metadata
+	if err := get(ctx, u, &metadata); err != nil {
+		return maven.Metadata{}, err
+	}
+
+	return metadata, nil
+}
+
+func get(ctx context.Context, url string, dst interface{}) error {
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
+	if err != nil {
+		return fmt.Errorf("failed to make new request: %w", err)
 	}
 
 	resp, err := http.DefaultClient.Do(req)
 	if err != nil {
-		return maven.Project{}, fmt.Errorf("%w: Maven registry query failed: %w", errAPIFailed, err)
+		return fmt.Errorf("%w: Maven registry query failed: %w", errAPIFailed, err)
 	}
 	defer resp.Body.Close()
 
 	if resp.StatusCode != http.StatusOK {
-		return maven.Project{}, fmt.Errorf("%w: Maven registry query status: %s", errAPIFailed, resp.Status)
+		return fmt.Errorf("%w: Maven registry query status: %s", errAPIFailed, resp.Status)
 	}
 
-	var proj maven.Project
-	if err := xml.NewDecoder(resp.Body).Decode(&proj); err != nil {
-		return maven.Project{}, fmt.Errorf("failed to decode Maven project: %w", err)
-	}
-
-	return proj, nil
+	return xml.NewDecoder(resp.Body).Decode(dst)
 }



Test patch:
diff --git a/internal/resolution/datasource/maven_registry_test.go b/internal/resolution/datasource/maven_registry_test.go
new file mode 100644
index 00000000..fa09cb43
--- /dev/null
+++ b/internal/resolution/datasource/maven_registry_test.go
@@ -0,0 +1,207 @@
+package datasource
+
+import (
+	"context"
+	"reflect"
+	"testing"
+
+	"deps.dev/util/maven"
+	"github.com/google/osv-scanner/internal/testutility"
+)
+
+func TestGetProject(t *testing.T) {
+	t.Parallel()
+
+	srv := testutility.NewMockHTTPServer(t)
+	client := &MavenRegistryAPIClient{
+		registry: srv.URL,
+	}
+	srv.SetResponse(t, "org/example/x.y.z/1.0.0/x.y.z-1.0.0.pom", []byte(`
+	<project>
+	  <groupId>org.example</groupId>
+	  <artifactId>x.y.z</artifactId>
+	  <version>1.0.0</version>
+	</project>
+	`))
+
+	got, err := client.GetProject(context.Background(), "org.example", "x.y.z", "1.0.0")
+	if err != nil {
+		t.Fatalf("failed to get Maven project %s:%s verion %s: %v", "org.example", "x.y.z", "1.0.0", err)
+	}
+	want := maven.Project{
+		ProjectKey: maven.ProjectKey{
+			GroupID:    "org.example",
+			ArtifactID: "x.y.z",
+			Version:    "1.0.0",
+		},
+	}
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("GetProject(%s, %s, %s):\ngot %v\nwant %v\n", "org.example", "x.y.z", "1.0.0", got, want)
+	}
+}
+
+func TestGetProjectSnapshot(t *testing.T) {
+	t.Parallel()
+
+	srv := testutility.NewMockHTTPServer(t)
+	client := &MavenRegistryAPIClient{
+		registry: srv.URL,
+	}
+	srv.SetResponse(t, "org/example/x.y.z/3.3.1-SNAPSHOT/maven-metadata.xml", []byte(`
+	<metadata>
+	  <groupId>org.example</groupId>
+	  <artifactId>x.y.z</artifactId>
+	  <versioning>
+	  <snapshot>
+	    <timestamp>20230302.052731</timestamp>
+	    <buildNumber>9</buildNumber>
+	  </snapshot>
+	  <lastUpdated>20230302052731</lastUpdated>
+	  <snapshotVersions>
+	    <snapshotVersion>
+	      <extension>jar</extension>
+	      <value>3.3.1-20230302.052731-9</value>
+	      <updated>20230302052731</updated>
+	    </snapshotVersion>
+	    <snapshotVersion>
+	      <extension>pom</extension>
+	      <value>3.3.1-20230302.052731-9</value>
+	      <updated>20230302052731</updated>
+	    </snapshotVersion>
+	  </snapshotVersions>
+	  </versioning>
+	</metadata>
+	`))
+	srv.SetResponse(t, "org/example/x.y.z/3.3.1-SNAPSHOT/x.y.z-3.3.1-20230302.052731-9.pom", []byte(`
+	<project>
+	  <groupId>org.example</groupId>
+	  <artifactId>x.y.z</artifactId>
+	  <version>3.3.1-SNAPSHOT</version>
+	</project>
+	`))
+
+	got, err := client.GetProject(context.Background(), "org.example", "x.y.z", "3.3.1-SNAPSHOT")
+	if err != nil {
+		t.Fatalf("failed to get Maven project %s:%s verion %s: %v", "org.example", "x.y.z", "3.3.1-SNAPSHOT", err)
+	}
+	want := maven.Project{
+		ProjectKey: maven.ProjectKey{
+			GroupID:    "org.example",
+			ArtifactID: "x.y.z",
+			Version:    "3.3.1-SNAPSHOT",
+		},
+	}
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("GetProject(%s, %s, %s):\ngot %v\nwant %v\n", "org.example", "x.y.z", "3.3.1-SNAPSHOT", got, want)
+	}
+}
+
+func TestGetArtifactMetadata(t *testing.T) {
+	t.Parallel()
+
+	srv := testutility.NewMockHTTPServer(t)
+	client := &MavenRegistryAPIClient{
+		registry: srv.URL,
+	}
+	srv.SetResponse(t, "org/example/x.y.z/maven-metadata.xml", []byte(`
+	<metadata>
+	  <groupId>org.example</groupId>
+	  <artifactId>x.y.z</artifactId>
+	  <versioning>
+	    <latest>3.0</latest>
+	    <release>3.0</release>
+	    <versions>
+	      <version>1.0</version>
+	      <version>2.0</version>
+		  <version>3.0</version>
+	    </versions>
+	  </versioning>
+	</metadata>
+	`))
+
+	got, err := client.GetArtifactMetadata(context.Background(), "org.example", "x.y.z")
+	if err != nil {
+		t.Fatalf("failed to get artifact metadata for %s:%s: %v", "org.example", "x.y.z", err)
+	}
+	want := maven.Metadata{
+		GroupID:    "org.example",
+		ArtifactID: "x.y.z",
+		Versioning: maven.Versioning{
+			Latest:  "3.0",
+			Release: "3.0",
+			Versions: []maven.String{
+				"1.0",
+				"2.0",
+				"3.0",
+			},
+		},
+	}
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("GetArtifactMetadata(%s, %s):\ngot %v\nwant %v\n", "org.example", "x.y.z", got, want)
+	}
+}
+
+func TestGetVersionMetadata(t *testing.T) {
+	t.Parallel()
+
+	srv := testutility.NewMockHTTPServer(t)
+	client := &MavenRegistryAPIClient{
+		registry: srv.URL,
+	}
+	srv.SetResponse(t, "org/example/x.y.z/3.3.1-SNAPSHOT/maven-metadata.xml", []byte(`
+	<metadata>
+	  <groupId>org.example</groupId>
+	  <artifactId>x.y.z</artifactId>
+	  <versioning>
+	  <snapshot>
+	    <timestamp>20230302.052731</timestamp>
+	    <buildNumber>9</buildNumber>
+	  </snapshot>
+	  <lastUpdated>20230302052731</lastUpdated>
+	  <snapshotVersions>
+	    <snapshotVersion>
+	      <extension>jar</extension>
+	      <value>3.3.1-20230302.052731-9</value>
+	      <updated>20230302052731</updated>
+	    </snapshotVersion>
+	    <snapshotVersion>
+	      <extension>pom</extension>
+	      <value>3.3.1-20230302.052731-9</value>
+	      <updated>20230302052731</updated>
+	    </snapshotVersion>
+	  </snapshotVersions>
+	  </versioning>
+	</metadata>
+	`))
+
+	got, err := client.getVersionMetadata(context.Background(), "org.example", "x.y.z", "3.3.1-SNAPSHOT")
+	if err != nil {
+		t.Fatalf("failed to get metadata for %s:%s verion %s: %v", "org.example", "x.y.z", "3.3.1-SNAPSHOT", err)
+	}
+	want := maven.Metadata{
+		GroupID:    "org.example",
+		ArtifactID: "x.y.z",
+		Versioning: maven.Versioning{
+			Snapshot: maven.Snapshot{
+				Timestamp:   "20230302.052731",
+				BuildNumber: 9,
+			},
+			LastUpdated: "20230302052731",
+			SnapshotVersions: []maven.SnapshotVersion{
+				{
+					Extension: "jar",
+					Value:     "3.3.1-20230302.052731-9",
+					Updated:   "20230302052731",
+				},
+				{
+					Extension: "pom",
+					Value:     "3.3.1-20230302.052731-9",
+					Updated:   "20230302052731",
+				},
+			},
+		},
+	}
+	if !reflect.DeepEqual(got, want) {
+		t.Errorf("getVersionMetadata(%s, %s):\ngot %v\nwant %v\n", "org.example", "x.y.z", got, want)
+	}
+}
diff --git a/internal/resolution/datasource/maven_test.go b/internal/resolution/datasource/maven_test.go
deleted file mode 100644
index 6a25a14e..00000000
--- a/internal/resolution/datasource/maven_test.go
+++ /dev/null
@@ -1,41 +0,0 @@
-package datasource
-
-import (
-	"context"
-	"reflect"
-	"testing"
-
-	"deps.dev/util/maven"
-	"github.com/google/osv-scanner/internal/testutility"
-)
-
-func TestGetProject(t *testing.T) {
-	t.Parallel()
-
-	srv := testutility.NewMockHTTPServer(t)
-	client := &MavenRegistryAPIClient{
-		registry: srv.URL,
-	}
-	srv.SetResponse(t, "org/example/x.y.z/1.0.0/x.y.z-1.0.0.pom", []byte(`
-	<project>
-	  <groupId>org.example</groupId>
-	  <artifactId>x.y.z</artifactId>
-	  <version>1.0.0</version>
-	</project>
-	`))
-
-	got, err := client.GetProject(context.Background(), "org.example", "x.y.z", "1.0.0")
-	if err != nil {
-		t.Fatalf("failed to get Maven project %s:%s verion %s: %v", "org.example", "x.y.z", "1.0.0", err)
-	}
-	want := maven.Project{
-		ProjectKey: maven.ProjectKey{
-			GroupID:    "org.example",
-			ArtifactID: "x.y.z",
-			Version:    "1.0.0",
-		},
-	}
-	if !reflect.DeepEqual(got, want) {
-		t.Errorf("GetProject(%s, %s, %s):\ngot %v\nwant %v\n", "org.example", "x.y.z", "1.0.0", got, want)
-	}
-}