Repository: p2panda/aquadoggo (language: rust)

Issue:
Materialised blobs sometimes corrupted / missing bytes
Sometimes blobs are materialised with missing bytes. Noticed mostly when uploading larger files.




Discussion / hints:
I wonder if it's connected to https://github.com/p2panda/aquadoggo/issues as we use the pagination API to retrieve blob pieces.
Hmn, looking at the pictures I doubt that, the data is there, its just corrupted? If it would be the pagination issues we would have whole parts missing?
Not sure about the images, but I've observed materialised blobs (uploaded successfully using `send-to-node`) having a smaller file size (roughly 40mb upload losing a couple of mb). I assume because it's missing pieces. I'll try and write a test for this though to support the theory.
Yeah, with large blobs it's possible as pagination will play a role then.

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 123ea7d19..2bcc08d91 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -37,6 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Do not panic when blob does not have all pieces yet [#563](https://github.com/p2panda/aquadoggo/pull/563)
 - Fix `blob` tasks being triggered too often [#578](https://github.com/p2panda/aquadoggo/pull/578)
 - Fix `schema` tasks being triggered too often [#581](https://github.com/p2panda/aquadoggo/pull/581)
+- Fix blobs getting corrupted when written to the file system [#587](https://github.com/p2panda/aquadoggo/pull/587)
 
 ## [0.5.0]
 
diff --git a/aquadoggo/src/materializer/tasks/blob.rs b/aquadoggo/src/materializer/tasks/blob.rs
index 9503abc8f..28f76c558 100644
--- a/aquadoggo/src/materializer/tasks/blob.rs
+++ b/aquadoggo/src/materializer/tasks/blob.rs
@@ -93,7 +93,7 @@ pub async fn blob_task(context: Context, input: TaskInput) -> TaskResult<TaskInp
 
             while let Some(value) = stream.next().await {
                 match value {
-                    Ok(buf) => file.write(&buf).await.map_err(|err| {
+                    Ok(buf) => file.write_all(&buf).await.map_err(|err| {
                         TaskError::Critical(format!(
                             "Error occurred when writing to blob file @ {}: {}",
                             blob_view_path.display(),
@@ -172,6 +172,46 @@ mod tests {
         })
     }
 
+    #[rstest]
+    fn materializes_larger_blob_to_filesystem(key_pair: KeyPair) {
+        test_runner(|mut node: TestNode| async move {
+            // Publish blob
+            let length = 10e6 as u32; // 5MB
+            let blob_data: Vec<u8> = (0..length).map(|_| rand::random::<u8>()).collect();
+            let blob_view_id = add_blob(
+                &mut node,
+                &blob_data,
+                256 * 1000,
+                "application/octet-stream",
+                &key_pair,
+            )
+            .await;
+
+            // Run blob task
+            let result = blob_task(
+                node.context.clone(),
+                TaskInput::DocumentViewId(blob_view_id.clone()),
+            )
+            .await;
+
+            // It shouldn't fail
+            assert!(result.is_ok());
+            // It should return no extra tasks
+            assert!(result.unwrap().is_none());
+
+            // Construct the expected path to the blob view file
+            let base_path = &node.context.config.blobs_base_path;
+            let blob_path = base_path.join(blob_view_id.to_string());
+
+            // Read from this file
+            let retrieved_blob_data = fs::read(blob_path).await;
+
+            // Number of bytes for the publish and materialized blob should be the same
+            assert!(retrieved_blob_data.is_ok());
+            assert_eq!(blob_data.len(), retrieved_blob_data.unwrap().len());
+        })
+    }
+
     #[rstest]
     fn rejects_incorrect_schema(key_pair: KeyPair) {
         test_runner(|mut node: TestNode| async move {


Test patch: