Repository: Azure/azure-sdk-for-go (Go)

Issue:
Cosmos Go SDK fails to handle MaxInt64 in the QueryItemsResponse
### Bug Report

- import path of package in question: `github.com/Azure/azure-sdk-for-go/sdk/data/azcosmos`
- SDK version: `github.com/Azure/azure-sdk-for-go/sdk/data/azcosmos v0.3.6`
- output of `go version`: `go version go1.21.7 darwin/amd64`

#### What happened?
- When data persisted in cosmos db has `math.MaxInt64` (9223372036854775807), the query response will return the wrong number (9223372036854776000)
- Note that this is only an bug for query, not for point get (`ReadItem`)
#### What did you expect or want to happen?
- The data should be returned as it is and not modified by the sdk
#### How can we reproduce it?
- Write some data that has a field with value 9223372036854775807
- Issue a query that utilizes `NewQueryItemsPager` [method](https://github.com/Azure/azure-sdk-for-go/blob/bc4f15f7e8b1452176119679bb9cbcb205afaa86/sdk/data/azcosmos/cosmos_container.go#L409) 
- The response will return the object with a different value 9223372036854776000
#### Where is the bug?
- The bug lies in the logic trying to [unmarshal query response to Json](https://github.com/Azure/azure-sdk-for-go/blob/986b63c9c081319c830a0ce6e4a19dcc8fc37c1f/sdk/data/azcosmos/cosmos_query_response.go#L42-L45) 
- When a field is set as max int64 9223372036854775807, it will be converted to 9223372036854776000, because of json.Unmarshal will convert it to float64.  See [json package - encoding/json - Go Packages](https://pkg.go.dev/encoding/json#Unmarshal)


Discussion:
Please see https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits#per-item-limits

![image](https://github.com/Azure/azure-sdk-for-go/assets/1633401/f558688d-4239-4c92-aa43-1fa3ad43e14b)

`9223372036854775807` seems outside of the safe boundary of [IEEE754](https://www.rfc-editor.org/rfc/rfc8259#ref-IEEE754) (9007199254740992) to work with the service. This will run into issues across other areas (like Portal interactions or other languages).

![image](https://github.com/Azure/azure-sdk-for-go/assets/1633401/86eda2ce-4b98-43aa-a083-624dfd0606fe)

While in Go this could be fixed by removing the unmarshal call and change it to something else, the issue will still appear in other areas.

Storing these numbers as strings instead would be the recommendation. 

We can certainly modify the Go SDK but that would not remove the problem across other service interactions in other languages/scenarios.

@ealsur Thanks for taking a look and providing the detailed docs! You are right, we also see the number handling issue with Azure Cosmos Portal, but the underlining data remains correct.

> We can certainly modify the Go SDK but that would not remove the problem across other service interactions in other languages/scenarios.

Do you think you can introduce some option to configure? This is how spanner does it https://github.com/googleapis/google-cloud-go/blob/main/spanner/value.go#L120-L132
Hi Matias,

Thanks for taking a look and providing the detailed docs! You are right, we
also see the number handling issue with Azure Cosmos Portal, but the
underlying data remains correct.

We can certainly modify the Go SDK but that would not remove the problem
> across other service interactions in other languages/scenarios.


Do you think you can introduce some option to configure? This is how
spanner does it
https://github.com/googleapis/google-cloud-go/blob/main/spanner/value.go#L120-L132

Best,
Xin


On Tue, Mar 5, 2024 at 1:09 PM Matias Quaranta ***@***.***>
wrote:

> Please see
> https://learn.microsoft.com/en-us/azure/cosmos-db/concepts-limits#per-item-limits
>
> image.png (view on web)
> <https://github.com/Azure/azure-sdk-for-go/assets/1633401/f558688d-4239-4c92-aa43-1fa3ad43e14b>
>
> 9223372036854775807 seems outside of the safe boundary of IEEE754
> <https://www.rfc-editor.org/rfc/rfc8259#ref-IEEE754> (9007199254740992)
> to work with the service. This will run into issues across other areas
> (like Portal interactions or other languages).
>
> image.png (view on web)
> <https://github.com/Azure/azure-sdk-for-go/assets/1633401/86eda2ce-4b98-43aa-a083-624dfd0606fe>
>
> While in Go this could be fixed by removing the unmarshal call and change
> it to something else, the issue will still appear in other areas.
>
> Storing these numbers as strings instead would be the recommendation.
>
> We can certainly modify the Go SDK but that would not remove the problem
> across other service interactions in other languages/scenarios.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/Azure/azure-sdk-for-go/issues/22523#issuecomment-1979639671>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A2AOMRJUGOXFVTYMK3I6CRTYWYYBRAVCNFSM6AAAAABEHYHOJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZZGYZTSNRXGE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

> the underlining data remains correct.

Right, the problem is that it is not consistent across. That's why it's safer to not use them.

> Do you think you can introduce some option to configure?

One solution would be to simply remove the unmarshal and attempt to simply return the byte[] of the document (unsure exactly what the implementation would look like right now). If that can be done (like in ReadItem), then at least from the Go SDK perspective, you'd be ok. But the problem is still there.

In the future when things like Cross Partition Query support is added with things like aggregations (that need to happen on the client), then this will probably happen again.

For Json serialization, our Cosmos SDK uses `azruntime.UnmarshalAsJSON`, which uses `json.Unmarshal` under the covers. I don't know if `"encoding/json"` has any such setting that you could customize.
In the `encoding/json` package, there are `decoder` and `unmarshaller`, the formal gives you a little bit more control.  
For example, it has https://pkg.go.dev/encoding/json#Decoder.UseNumber, which config the Decoder to unmarshal a number into an interface{} as a [Number](https://pkg.go.dev/encoding/json#Number) instead of as a float64.
Thanks @CurryFishBalls9527!

@ealsur Do you think this is something you can introduce to the sdk?
I can add it to the backlog. As I mentioned, right now our SDK does not invoke `json` directly but consumes it from `azruntime`. 

But even if we do this, please see my other comments, using these Numbers will have further potential repercussions outside of this SDK.



Patch:
diff --git a/sdk/data/azcosmos/cosmos_query_response.go b/sdk/data/azcosmos/cosmos_query_response.go
index 53db1321fe5a..50ca17fb0363 100644
--- a/sdk/data/azcosmos/cosmos_query_response.go
+++ b/sdk/data/azcosmos/cosmos_query_response.go
@@ -58,7 +58,7 @@ func newQueryResponse(resp *http.Response) (QueryItemsResponse, error) {
 }
 
 type queryServiceResponse struct {
-	Documents []any `json:"Documents,omitempty"`
+	Documents []json.RawMessage `json:"Documents,omitempty"`
 }
 
 // QueryContainersResponse contains response from the container query operation.


Test patch:
diff --git a/sdk/data/azcosmos/cosmos_query_response_test.go b/sdk/data/azcosmos/cosmos_query_response_test.go
index db8d70350d14..2d8beacbd7f6 100644
--- a/sdk/data/azcosmos/cosmos_query_response_test.go
+++ b/sdk/data/azcosmos/cosmos_query_response_test.go
@@ -95,6 +95,80 @@ func TestQueryResponseParsing(t *testing.T) {
 	}
 }
 
+func TestQueryResponseParsingWithMaxInt64(t *testing.T) {
+	queryResponseRaw := map[string][]map[string]interface{}{
+		"Documents": {
+			{"id": "id1", "name": "name", "maxInt64": 9223372036854775807},
+			{"id": "id2", "name": "name", "maxInt64": 9223372036854775807},
+		},
+	}
+
+	jsonString, err := json.Marshal(queryResponseRaw)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	srv, close := mock.NewTLSServer()
+	defer close()
+	srv.SetResponse(
+		mock.WithBody(jsonString),
+		mock.WithHeader(cosmosHeaderEtag, "someEtag"),
+		mock.WithHeader(cosmosHeaderQueryMetrics, "someQueryMetrics"),
+		mock.WithHeader(cosmosHeaderIndexUtilization, "indexUtilization"),
+		mock.WithHeader(cosmosHeaderActivityId, "someActivityId"),
+		mock.WithHeader(cosmosHeaderRequestCharge, "13.42"))
+
+	req, err := azruntime.NewRequest(context.Background(), http.MethodGet, srv.URL())
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	pl := azruntime.NewPipeline("azcosmostest", "v1.0.0", azruntime.PipelineOptions{}, &policy.ClientOptions{Transport: srv})
+	resp, _ := pl.Do(req)
+	parsedResponse, err := newQueryResponse(resp)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if parsedResponse.RawResponse == nil {
+		t.Fatal("parsedResponse.RawResponse is nil")
+	}
+
+	if parsedResponse.ActivityID != "someActivityId" {
+		t.Errorf("Expected ActivityId to be %s, but got %s", "someActivityId", parsedResponse.ActivityID)
+	}
+
+	if parsedResponse.RequestCharge != 13.42 {
+		t.Errorf("Expected RequestCharge to be %f, but got %f", 13.42, parsedResponse.RequestCharge)
+	}
+
+	if parsedResponse.ETag != "someEtag" {
+		t.Errorf("Expected ETag to be %s, but got %s", "someEtag", parsedResponse.ETag)
+	}
+
+	if *parsedResponse.QueryMetrics != "someQueryMetrics" {
+		t.Errorf("Expected IndexMetrics to be %s, but got %s", "someQueryMetrics", *parsedResponse.IndexMetrics)
+	}
+
+	if *parsedResponse.IndexMetrics != "indexUtilization" {
+		t.Errorf("Expected IndexUtilization to be %s, but got %s", "indexUtilization", *parsedResponse.IndexMetrics)
+	}
+
+	if len(parsedResponse.Items) != 2 {
+		t.Errorf("Expected 2 documents, but got %d", len(parsedResponse.Items))
+	}
+
+	firstItem := "{\"id\":\"id1\",\"maxInt64\":9223372036854775807,\"name\":\"name\"}"
+	if string(parsedResponse.Items[0]) != firstItem {
+		t.Errorf("Expected first item to be %s, but got %s", firstItem, parsedResponse.Items[0])
+	}
+
+	secondItem := "{\"id\":\"id2\",\"maxInt64\":9223372036854775807,\"name\":\"name\"}"
+	if string(parsedResponse.Items[1]) != secondItem {
+		t.Errorf("Expected second item to be %s, but got %s", secondItem, parsedResponse.Items[1])
+	}
+}
+
 func TestQueryResponseValueParsing(t *testing.T) {
 	queryResponseRaw := map[string][]string{
 		"Documents": {"id1", "id2"},