Repository: google/osv.dev

Issue:
Bulk query API
A bulk query API would allow developers to more easily query the API without hitting rate limits. It would also help with scenarios like #257, where an SBOM will contain many dependencies.


Discussion:
@JamieMagee  Not sure if this helps answer your question but we had a similar requirement (Issue - https://github.com/google/osv/issues/306)
FWIW, we went with using the GCS Bucket option and do version comparison on our end. Would be nice to leverage the API without hitting the rate limit but for now just downloading the zip and using that.

Yes, I went a similar route. I wrote a package, https://github.com/JamieMagee/osv-offline, to do this in Node.js. It's to support [this issue](https://github.com/renovatebot/renovate/issues/6562) in Renovate
Sweet, will check it out.
Also, renovate looks pretty interesting thanks for linking it as well.
Thanks a lot for filing this issue and explaining the use case! This is definitely something we want to support, to make it easier for our users to consume the data.

Note that despite our documentation, there are actually currently no rate limits enforced on the endpoints at https://api.osv.dev (barring detected abuse/DoS attempts), and definitely no plans to have rate limits on the `/vulns/ID` GET endpoint in the future. 

Just to make sure we're on the same page, we would support something like:

```json
{
  "query": [
    { "package": { "ecosystem": "blah", "name": "blah" }, "version": "1.0.0"},
    { "package": { "ecosystem": "blah", "name": "blah2" }, "version": "1.0.0" },
    { "package": { "ecosystem": "blah", "name": "blah3" }, "version": "1.0.0" },
  ]
}
```

And responses would look something like like:
```json
{
  "results": [
    {"vulns": []}, 
    {"vulns": [ {"id": "GHSA-BLAH"}, {"id": "PYSEC-BLAH"}, "id": "OSV-BLAH" ] },
    {"vulns": [ {"id": "OSV-BLAH2"}, {"id": "RUSTSEC-BLAH"} ] },
  ]
}
```

Would this work for your use case?

Note that we may not be able to return the full details of all vulnerabilities inline in the response here due to the potential volume. To get the details of the individual vulnerabilities, you'd have to call the `/vulns/ID` GET endpoint (which as previously stated does not/will not have any rate limits).
Yeah, returning a minimal, dehydrated, response is reasonable.

Thanks for clarifying about the rate limits. I've gone the route of creating an offline copy of the database for use in Renovate for now, because individual queries for each dependency in a project would be too much of a performance hit. You know how large the dependency graphs can get for node projects 😅  We could probably migrate to a bulk query API, but we can cross that bridge when it comes to it.
Thanks for clarifying @JamieMagee  FYI we're looking to implement this in the next few weeks!
The implementation of this has started in #418. 
This is deployed to our staging API instance at `https://api-staging.osv.dev/v1/querybatch`.

Sample query:

```json                                                                                                                                                                                                                                                                        
{
  "queries": [
    {
      "package": {
        "purl": "pkg:pypi/antlr4-python3-runtime@4.7.2"
      }
    },
    {
      "commit": "6879efc2c1596d11a6a6ad296f80063b558d5e0f"
    },
    {
      "package": {
        "ecosystem": "PyPI",
        "name": "jinja2"
      },
      "version": "2.4.1"
    }
  ]
}
```
@JamieMagee and others: Would you be willing to try the staging API to see if this would fit your needs? 
I tried the API in staging and it definitely fits our needs assuming the response ordering is guaranteed to match the request. It seems like it is.
> I tried the API in staging and it definitely fits our needs assuming the response ordering is guaranteed to match the request. It seems like it is.

That's right. The response ordering will be guaranteed to match the input. 
This is now also deployed to our production instance at https://api.osv.dev/v1/querybatch.

Resolving patch:
diff --git a/docs/build.py b/docs/build.py
--- a/docs/build.py
+++ b/docs/build.py
@@ -76,7 +76,7 @@ def main():
   spec['tags'] = [{
       'name': 'api',
       'x-displayName': 'API',
-      'description': 'The API has 2 methods:'
+      'description': 'The API has 3 methods:'
   }, {
       'name': 'vulnerability_schema',
       'x-displayName': 'Vulnerability schema',
@@ -93,6 +93,7 @@ def main():
   }]
 
   spec['paths']['/v1/query']['post']['tags'] = ['api']
+  spec['paths']['/v1/querybatch']['post']['tags'] = ['api']
   spec['paths']['/v1/vulns/{id}']['get']['tags'] = ['api']
 
   spec['paths']['/v1/query']['post']['x-code-samples'] = [{
@@ -107,6 +108,33 @@ def main():
            '  "https://api.osv.dev/v1/query"')
   }]
 
+  spec['paths']['/v1/querybatch']['post']['x-code-samples'] = [{
+      'lang':
+          'Curl example',
+      'source':
+          ("""cat <<EOF | curl -X POST -d @- "https://api.osv.dev/v1/querybatch"
+{
+  "queries": [
+    {
+      "package": {
+        "purl": "pkg:pypi/antlr4-python3-runtime@4.7.2"
+      }
+    },
+    {
+      "commit": "6879efc2c1596d11a6a6ad296f80063b558d5e0f"
+    },
+    {
+      "package": {
+        "ecosystem": "PyPI",
+        "name": "jinja2"
+      },
+      "version": "2.4.1"
+    }
+  ]
+}
+EOF""")
+  }]
+
   spec['paths']['/v1/vulns/{id}']['get']['x-code-samples'] = [{
       'lang': 'Curl example',
       'source': 'curl "https://api.osv.dev/v1/vulns/OSV-2020-111"'
diff --git a/gcp/api/osv_service_v1_pb2_grpc.py b/gcp/api/osv_service_v1_pb2_grpc.py
--- a/gcp/api/osv_service_v1_pb2_grpc.py
+++ b/gcp/api/osv_service_v1_pb2_grpc.py
@@ -53,8 +53,7 @@ def QueryAffected(self, request, context):
         raise NotImplementedError('Method not implemented!')
 
     def QueryAffectedBatch(self, request, context):
-        """Query vulnerabilities for a particular project at given commits and
-        versions.
+        """Query vulnerabilities (batched) for given package versions and commits.
         """
         context.set_code(grpc.StatusCode.UNIMPLEMENTED)
         context.set_details('Method not implemented!')


Test patch: