Repository: ramsayleung/rspotify
Pull request #526: Fix: deserialization of pagination with null items
Author: login: DimitriTimoz
Created: 2025-05-26T18:39:04Z | Merged: 2025-05-29T20:46:47Z

## Description

Fix deserialization panic on the search endpoint when the API returns a list of elements containing one or more null values. Also fix CI (clippy warnings).

## Motivation and Context
When the Spotify API returns results for a search query, sometimes items are null, which causes a panic during deserialization.
Here is an example of a case that causes a panic:
```json
{
    "playlists": {
        "href": "https://api.spotify.com/v1/search?offset=17&limit=2&query=My%20Playlist&type=playlist&market=FR",
        "limit": 2,
        "next": "https://api.spotify.com/v1/search?offset=19&limit=2&query=My%20Playlist&type=playlist&market=FR",
        "offset": 17,
        "previous": "https://api.spotify.com/v1/search?offset=15&limit=2&query=My%20Playlist&type=playlist&market=FR",
        "total": 901,
        "items": [
            {
                ...
            },
            null // <--
        ]
    }
}
```

## Type of change

- [x] Bug fix (non-breaking change which fixes an issue)

## How has this been tested?

I performed a search query on the FR market for playlists with the query "My Playlist". The response contained a null item. After applying the fix, I checked the same query again and confirmed that the panic no longer occurs (the "null" is still present in the response, but does not cause any error).




Diff:
diff --git a/rspotify-model/src/auth.rs b/rspotify-model/src/auth.rs
index d21c6607..c4896852 100644
--- a/rspotify-model/src/auth.rs
+++ b/rspotify-model/src/auth.rs
@@ -84,9 +84,8 @@ impl Token {
     /// is how much a request would take in the worst case scenario).
     #[must_use]
     pub fn is_expired(&self) -> bool {
-        self.expires_at.map_or(true, |expiration| {
-            Utc::now() + TimeDelta::try_seconds(10).unwrap() >= expiration
-        })
+        self.expires_at
+            .is_none_or(|expiration| Utc::now() + TimeDelta::try_seconds(10).unwrap() >= expiration)
     }
 
     /// Generates an HTTP token authorization header with proper formatting
diff --git a/rspotify-model/src/page.rs b/rspotify-model/src/page.rs
index f052a639..9912ac46 100644
--- a/rspotify-model/src/page.rs
+++ b/rspotify-model/src/page.rs
@@ -1,16 +1,29 @@
 //! All kinds of page object
+use serde::{de::DeserializeOwned, Deserialize, Serialize};
 
-use serde::{Deserialize, Serialize};
+/// Custom deserializer to handle `Vec<Option<T>>` and filter out `None` values
+/// This is useful for deserializing lists that may contain null values that are not relevants
+fn vec_without_nulls<'de, D, T>(deserializer: D) -> Result<Vec<T>, D::Error>
+where
+    T: serde::Deserialize<'de>,
+    D: serde::Deserializer<'de>,
+{
+    let v = Vec::<Option<T>>::deserialize(deserializer)?;
+    Ok(v.into_iter().flatten().collect())
+}
 
 /// Paging object
 #[derive(Clone, Debug, Serialize, Deserialize, PartialEq, Eq, Default)]
-pub struct Page<T> {
+pub struct Page<T: DeserializeOwned> {
     pub href: String,
+    #[serde(deserialize_with = "vec_without_nulls")]
     pub items: Vec<T>,
     pub limit: u32,
     pub next: Option<String>,
     pub offset: u32,
     pub previous: Option<String>,
+    /// This field could mismatch the actual number of items in `items` field
+    /// because sometimes the API returns `null` items that are not included in the `items` field.
     pub total: u32,
 }
 
diff --git a/src/clients/pagination/iter.rs b/src/clients/pagination/iter.rs
index 692e06a3..ee2de1e1 100644
--- a/src/clients/pagination/iter.rs
+++ b/src/clients/pagination/iter.rs
@@ -1,11 +1,12 @@
 //! Synchronous implementation of automatic pagination requests.
 
 use crate::{model::Page, ClientError, ClientResult};
+use serde::de::DeserializeOwned;
 
 /// Alias for `Iterator<Item = T>`, since sync mode is enabled.
 pub type Paginator<'a, T> = Box<dyn Iterator<Item = T> + 'a>;
 
-pub fn paginate_with_ctx<'a, Ctx: 'a, T: 'a, Request>(
+pub fn paginate_with_ctx<'a, Ctx: 'a, T: DeserializeOwned + 'a, Request>(
     ctx: Ctx,
     req: Request,
     page_size: u32,
@@ -17,7 +18,10 @@ where
 }
 
 /// This is used to handle paginated requests automatically.
-pub fn paginate<'a, T: 'a, Request>(req: Request, page_size: u32) -> Paginator<'a, ClientResult<T>>
+pub fn paginate<'a, T: DeserializeOwned + 'a, Request>(
+    req: Request,
+    page_size: u32,
+) -> Paginator<'a, ClientResult<T>>
 where
     Request: 'a + Fn(u32, u32) -> ClientResult<Page<T>>,
 {
@@ -40,7 +44,7 @@ struct PageIterator<Request> {
     page_size: u32,
 }
 
-impl<T, Request> Iterator for PageIterator<Request>
+impl<T: DeserializeOwned, Request> Iterator for PageIterator<Request>
 where
     Request: Fn(u32, u32) -> ClientResult<Page<T>>,
 {
diff --git a/src/clients/pagination/stream.rs b/src/clients/pagination/stream.rs
index bbf3aacf..360d013f 100644
--- a/src/clients/pagination/stream.rs
+++ b/src/clients/pagination/stream.rs
@@ -5,6 +5,7 @@ use crate::{model::Page, ClientResult};
 use std::pin::Pin;
 
 use futures::{future::Future, stream::Stream};
+use serde::de::DeserializeOwned;
 
 /// Alias for `futures::stream::Stream<Item = T>`, since async mode is enabled.
 pub type Paginator<'a, T> = Pin<Box<dyn Stream<Item = T> + 'a + Send>>;
@@ -18,7 +19,7 @@ pub fn paginate_with_ctx<'a, Ctx: 'a + Send, T, Request>(
     page_size: u32,
 ) -> Paginator<'a, ClientResult<T>>
 where
-    T: 'a + Unpin + Send,
+    T: 'a + Unpin + Send + DeserializeOwned,
     Request: 'a + for<'ctx> Fn(&'ctx Ctx, u32, u32) -> RequestFuture<'ctx, T> + Send,
 {
     use async_stream::stream;
@@ -46,7 +47,7 @@ where
 
 pub fn paginate<'a, T, Fut, Request>(req: Request, page_size: u32) -> Paginator<'a, ClientResult<T>>
 where
-    T: 'a + Unpin + Send,
+    T: 'a + Unpin + Send + DeserializeOwned,
     Fut: Future<Output = ClientResult<Page<T>>> + Send,
     Request: 'a + Fn(u32, u32) -> Fut + Send,
 {
diff --git a/src/clients/pagination/wasm_stream.rs b/src/clients/pagination/wasm_stream.rs
index 58df6dbd..9054f92e 100644
--- a/src/clients/pagination/wasm_stream.rs
+++ b/src/clients/pagination/wasm_stream.rs
@@ -5,6 +5,7 @@ use crate::{model::Page, ClientResult};
 use std::pin::Pin;
 
 use futures::{future::Future, stream::Stream};
+use serde::de::DeserializeOwned;
 
 /// Alias for `futures::stream::Stream<Item = T>`, since async mode is enabled.
 pub type Paginator<'a, T> = Pin<Box<dyn Stream<Item = T> + 'a>>;
@@ -18,7 +19,7 @@ pub fn paginate_with_ctx<'a, Ctx: 'a, T, Request>(
     page_size: u32,
 ) -> Paginator<'a, ClientResult<T>>
 where
-    T: 'a + Unpin,
+    T: DeserializeOwned + 'a + Unpin,
     Request: 'a + for<'ctx> Fn(&'ctx Ctx, u32, u32) -> RequestFuture<'ctx, T>,
 {
     use async_stream::stream;
@@ -40,7 +41,7 @@ where
 
 pub fn paginate<'a, T, Fut, Request>(req: Request, page_size: u32) -> Paginator<'a, ClientResult<T>>
 where
-    T: 'a + Unpin,
+    T: DeserializeOwned + 'a + Unpin,
     Fut: Future<Output = ClientResult<Page<T>>>,
     Request: 'a + Fn(u32, u32) -> Fut,
 {
diff --git a/tests/test_models.rs b/tests/test_models.rs
index 0d0d91e1..44be108b 100644
--- a/tests/test_models.rs
+++ b/tests/test_models.rs
@@ -1223,3 +1223,26 @@ fn test_null_id_in_tracklink() {
     assert!(linked_from.id.is_none());
     assert_eq!(linked_from.r#type, Type::Track);
 }
+
+#[test]
+#[wasm_bindgen_test]
+fn test_null_item_in_page() {
+    let json = r#"
+{
+  "href": "https://api.spotify.com/v1/search?offset=17&limit=2&query=My%20Playlist&type=playlist&market=FR",
+  "items": [
+    null
+  ],
+  "limit": 20,
+  "next": null,
+  "offset": 0,
+  "previous": null,
+  "total": 1
+}"#;
+    let page: Page<SimplifiedPlaylist> = deserialize(json);
+    assert_eq!(page.items.len(), 0);
+    assert_eq!(page.total, 1); // Because one element is null so ignored at deserialization but counted in total
+    assert_eq!(page.limit, 20);
+    assert!(page.next.is_none());
+    assert!(page.previous.is_none());
+}


Reviews:
user: login: ramsayleung
state: COMMENTED
author_association: OWNER
submitted_at: 2025-05-26T23:40:26Z
user: login: DimitriTimoz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-27T09:38:08Z
user: login: ramsayleung
state: COMMENTED
author_association: OWNER
submitted_at: 2025-05-28T01:31:14Z
user: login: ramsayleung
state: COMMENTED
author_association: OWNER
submitted_at: 2025-05-28T01:33:47Z
user: login: ramsayleung
state: APPROVED
author_association: OWNER
submitted_at: 2025-05-28T15:49:57Z

Review comments:
diff_hunk: @@ -1,11 +1,20 @@
 //! All kinds of page object
+use serde::{de::DeserializeOwned, Deserialize, Serialize};
 
-use serde::{Deserialize, Serialize};
+fn vec_without_nulls<'de, D, T>(deserializer: D) -> Result<Vec<T>, D::Error>
path: rspotify-model/src/page.rs
user: login: ramsayleung
body: Would you add a unit test for this function?
created_at: 2025-05-26T23:34:19Z
author_association: OWNER
diff_hunk: @@ -1,11 +1,20 @@
 //! All kinds of page object
+use serde::{de::DeserializeOwned, Deserialize, Serialize};
 
-use serde::{Deserialize, Serialize};
+fn vec_without_nulls<'de, D, T>(deserializer: D) -> Result<Vec<T>, D::Error>
path: rspotify-model/src/page.rs
user: login: ramsayleung
body: And could you also add a comment for this function to demonstrate why do we need this custom deserialization function?
created_at: 2025-05-26T23:39:36Z
author_association: OWNER
diff_hunk: @@ -1,11 +1,20 @@
 //! All kinds of page object
+use serde::{de::DeserializeOwned, Deserialize, Serialize};
 
-use serde::{Deserialize, Serialize};
+fn vec_without_nulls<'de, D, T>(deserializer: D) -> Result<Vec<T>, D::Error>
path: rspotify-model/src/page.rs
user: login: DimitriTimoz
body: I’m working on it, and I’m wondering how we handle the mismatch between the total number of results and the length of the `Vec`. `Null` elements are counted in the `total` field, but they are not present in the vector.


created_at: 2025-05-27T09:38:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1,11 +1,20 @@
 //! All kinds of page object
+use serde::{de::DeserializeOwned, Deserialize, Serialize};
 
-use serde::{Deserialize, Serialize};
+fn vec_without_nulls<'de, D, T>(deserializer: D) -> Result<Vec<T>, D::Error>
path: rspotify-model/src/page.rs
user: login: ramsayleung
body: We could accept this and add a doc to the `search` endpoint to demonstrate this anomaly behaviour. In my perspective, the Spotify server shouldn't return a `null` item, it doesn't make sense, and it requires all the clients and library to handle this weird design. What could the client do to handle `null` item except for filter them out.
created_at: 2025-05-28T01:31:14Z
author_association: OWNER
diff_hunk: @@ -1223,3 +1223,26 @@ fn test_null_id_in_tracklink() {
     assert!(linked_from.id.is_none());
     assert_eq!(linked_from.r#type, Type::Track);
 }
+
+#[test]
+#[wasm_bindgen_test]
+fn test_null_item_in_page() {
+    let json = r#"
+{
+  "href": "https://api.spotify.com/v1/search?offset=17&limit=2&query=My%20Playlist&type=playlist&market=FR",
+  "items": [
+    null
+  ],
+  "limit": 20,
+  "next": null,
+  "offset": 0,
+  "previous": null,
+  "total": 1
+}"#;
+    let page: Page<SimplifiedPlaylist> = deserialize(json);
+    assert_eq!(page.items.len(), 0);
+    assert_eq!(page.total-1, 0); // Because one element is null so ignore at deserialization but counted in total
path: tests/test_models.rs
user: login: ramsayleung
body: We could just assert the `page.total` equal `1` to indicate we don't change the `total` field at all.
created_at: 2025-05-28T01:33:34Z
author_association: OWNER

PR comments:
user: login: ramsayleung
created_at: 2025-05-26T23:40:13Z
body: Thanks for the fix, could you help to also fix failed GitHub Action jobs?
author_association: OWNER
user: login: ramsayleung
created_at: 2025-05-28T01:35:12Z
body: Even though I know the failed GitHub Action jobs are not relevant to this PR, but all the GitHub Action tasks are failed, would you fix them in this CR?
author_association: OWNER
user: login: DimitriTimoz
created_at: 2025-05-28T20:55:40Z
body: @ramsayleung  the CI should pass know, I should have run the commands locally before.
author_association: CONTRIBUTOR
user: login: ramsayleung
created_at: 2025-05-29T06:39:10Z
body: Unfortunately, a job still failed, you could quickly fix it by:

```sh
cargo fmt -- --emit files
```
author_association: OWNER
user: login: ramsayleung
created_at: 2025-05-29T20:47:03Z
body: Thanks for your contribution :)
author_association: OWNER