Repository: dai-shi/waku
Pull request #850 (2024-08-26 20:32:53)

[fix] File upload
I have a simple HTML form which submits a file to a server action via multipart form upload. From the client side, the file looks proper, but when the server action receives it, the file is much larger? Is there some intermediate processing of this file or improper handling in the library? 

client sent:
```
File {
  lastModified: 1724514738000
  name: "bara.jpeg"
  size: 2094593
  type: "image/jpeg"
}
```

server received:
```
File {
  size: 3715003,
  type: 'image/jpeg',
  name: 'bara.jpeg',
  lastModified: 1724698582443
}
```

I went through the code, and the bug largely was parsing the body as a string. Instead we have an array buffer so we can manipulate it more directly. It's possible it may still work as a string, but I believe the array buffer leaves more flexibility.

I've only tested directly in the application I've been building

vercel[bot]: [vc]: #0jk8JMVW7E7dTq6CI7sWT68++0JiUj+0A5LZUYZq8hA=:eyJpc01vbm9yZXBvIjp0cnVlLCJ0eXBlIjoiZ2l0aHViIiwicHJvamVjdHMiOlt7Im5hbWUiOiJ3YWt1Iiwicm9vdERpcmVjdG9yeSI6bnVsbCwiaW5zcGVjdG9yVXJsIjoiaHR0cHM6Ly92ZXJjZWwuY29tL2RhaXNoaXMtcHJvamVjdHMvd2FrdS9EWUdLUUdqc1ZaVTNUUEFtR2NzVFhHZVRxd1JZIiwicHJldmlld1VybCI6Indha3UtZ2l0LWZvcmstY2pwYWlzLWNqcGFpcy1maXgtZmlsZS11cGxvYWQtZGFpc2hpcy1wcm9qZWN0cy52ZXJjZWwuYXBwIiwibmV4dENvbW1pdFN0YXR1cyI6IlBFTkRJTkcifV19
**The latest updates on your projects**. Learn more about [Vercel for Git ↗︎](https://vercel.link/github-learn-more)

| Name | Status | Preview | Updated (UTC) |
| :--- | :----- | :------ | :------ |
| **waku** | 🔄 Building ([Inspect](https://vercel.com/daishis-projects/waku/DYGKQGjsVZU3TPAmGcsTXGeTqwRY)) | [Visit Preview](https://waku-git-fork-cjpais-cjpais-fix-file-upload-daishis-projects.vercel.app) | Aug 26, 2024 8:32pm |


codesandbox-ci[bot]: This pull request is automatically built and testable in [CodeSandbox](https://codesandbox.io).

  To see build info of the built libraries, click [here](https://ci.codesandbox.io/status/dai-shi/waku/pr/850/builds/535460) or the icon next to each commit SHA.
cjpais: i would guess the right thing to do is probably move to busboy, etc
dai-shi: > i would guess the right thing to do is probably move to busboy, etc

We used to use busboy as React lib uses it, but it's Node.js specific and Waku now supports non-Node.js envs.
cjpais: tests are a good starting point, but really need more. large files, etc.. the current code does not pass. the tests themselves need some review for accuracy as well
cjpais: So I initially wrote this code as a hack to solve a specific issue for my project. I think after writing the tests and seeing some of the review comments, it might be nice to review this body of code in total and maybe write a better implementation to handle more cases (like binary, base64, quoted strings, etc). 

Mentioned in discord but I see Hono is doing this. Not sure if applicable given support for non node.js based runtimes, I'm not super familiar with the API's down to that level. But since Hono supports different runtimes, I imagine there are either polyfills or something to get the parsing working.

```
export const bufferToFormData = (
  arrayBuffer: ArrayBuffer,
  contentType: string
): Promise<FormData> => {
  const response = new Response(arrayBuffer, {
    headers: {
      'Content-Type': contentType,
    },
  })
  return response.formData()
}
```

if I get a chance today I may try something as simple as this. (however I am using bun runtime so it probably has support for the node.js apis)

Patch:
diff --git a/packages/waku/src/lib/renderers/rsc-renderer.ts b/packages/waku/src/lib/renderers/rsc-renderer.ts
index 496d3db9..d5ffea1a 100644
--- a/packages/waku/src/lib/renderers/rsc-renderer.ts
+++ b/packages/waku/src/lib/renderers/rsc-renderer.ts
@@ -9,9 +9,9 @@ import type {
 } from '../../server.js';
 import type { ResolvedConfig } from '../config.js';
 import { filePathToFileURL } from '../utils/path.js';
-import { parseFormData } from '../utils/form.js';
-import { streamToString } from '../utils/stream.js';
+import { streamToArrayBuffer } from '../utils/stream.js';
 import { decodeActionId } from '../renderers/utils.js';
+import { bufferToString, parseFormData } from '../utils/buffer.js';
 
 export const SERVER_MODULE_MAP = {
   'rsdw-server': 'react-server-dom-webpack/server.edge',
@@ -194,15 +194,16 @@ export async function renderRsc(
 
   let decodedBody: unknown | undefined = args.decodedBody;
   if (body) {
-    const bodyStr = await streamToString(body);
+    const bodyBuf = await streamToArrayBuffer(body);
     if (
       typeof contentType === 'string' &&
       contentType.startsWith('multipart/form-data')
     ) {
       // XXX This doesn't support streaming unlike busboy
-      const formData = parseFormData(bodyStr, contentType);
+      const formData = await parseFormData(bodyBuf, contentType);
       decodedBody = await decodeReply(formData, serverBundlerConfig);
-    } else if (bodyStr) {
+    } else if (bodyBuf.byteLength > 0) {
+      const bodyStr = bufferToString(bodyBuf);
       decodedBody = await decodeReply(bodyStr, serverBundlerConfig);
     }
   }
diff --git a/packages/waku/src/lib/utils/buffer.ts b/packages/waku/src/lib/utils/buffer.ts
new file mode 100644
index 00000000..7cd9c5ce
--- /dev/null
+++ b/packages/waku/src/lib/utils/buffer.ts
@@ -0,0 +1,16 @@
+export const parseFormData = async (
+  buffer: ArrayBuffer,
+  contentType: string,
+): Promise<FormData> => {
+  const response = new Response(buffer, {
+    headers: {
+      'content-type': contentType,
+    },
+  });
+  return response.formData();
+};
+
+export const bufferToString = (buffer: ArrayBuffer): string => {
+  const enc = new TextDecoder();
+  return enc.decode(buffer);
+};
diff --git a/packages/waku/src/lib/utils/form.ts b/packages/waku/src/lib/utils/form.ts
deleted file mode 100644
index a33c9214..00000000
--- a/packages/waku/src/lib/utils/form.ts
+++ /dev/null
@@ -1,33 +0,0 @@
-// TODO is this correct? better to use a library?
-export const parseFormData = (body: string, contentType: string) => {
-  const boundary = contentType.split('boundary=')[1];
-  const parts = body.split(`--${boundary}`);
-  const formData = new FormData();
-  for (const part of parts) {
-    if (part.trim() === '' || part === '--') continue;
-    const [rawHeaders, content] = part.split('\r\n\r\n', 2);
-    const headers = rawHeaders!.split('\r\n').reduce(
-      (acc, currentHeader) => {
-        const [key, value] = currentHeader.split(': ');
-        acc[key!.toLowerCase()] = value!;
-        return acc;
-      },
-      {} as Record<string, string>,
-    );
-    const contentDisposition = headers['content-disposition'];
-    const nameMatch = /name="([^"]+)"/.exec(contentDisposition!);
-    const filenameMatch = /filename="([^"]+)"/.exec(contentDisposition!);
-    if (nameMatch) {
-      const name = nameMatch[1];
-      if (filenameMatch) {
-        const filename = filenameMatch[1];
-        const type = headers['content-type'] || 'application/octet-stream';
-        const blob = new Blob([content!], { type });
-        formData.append(name!, blob, filename);
-      } else {
-        formData.append(name!, content!.trim());
-      }
-    }
-  }
-  return formData;
-};
diff --git a/packages/waku/src/lib/utils/stream.ts b/packages/waku/src/lib/utils/stream.ts
index d70ccc5a..5dfa4f15 100644
--- a/packages/waku/src/lib/utils/stream.ts
+++ b/packages/waku/src/lib/utils/stream.ts
@@ -11,6 +11,32 @@ export const concatUint8Arrays = (arrs: Uint8Array[]): Uint8Array => {
   return array;
 };
 
+// FIXME remove the two loops if possible, if it is more efficient
+export const streamToArrayBuffer = async (stream: ReadableStream) => {
+  const reader = stream.getReader();
+  const chunks = [];
+  let totalSize = 0;
+  let done = false;
+  let value: Uint8Array | undefined;
+
+  do {
+    ({ done, value } = await reader.read());
+    if (!done && value) {
+      chunks.push(value);
+      totalSize += value.length;
+    }
+  } while (!done);
+
+  const result = new Uint8Array(totalSize);
+  let offset = 0;
+  for (const chunk of chunks) {
+    result.set(chunk, offset);
+    offset += chunk.length;
+  }
+
+  return result.buffer;
+};
+
 export const streamToString = async (
   stream: ReadableStream,
 ): Promise<string> => {


Test patch:
diff --git a/packages/waku/tests/buffer.test.ts b/packages/waku/tests/buffer.test.ts
new file mode 100644
index 00000000..b99c8d28
--- /dev/null
+++ b/packages/waku/tests/buffer.test.ts
@@ -0,0 +1,338 @@
+import { describe, it, expect } from 'vitest';
+import { bufferToString, parseFormData } from '../src/lib/utils/buffer';
+
+// Minimal valid 1x1 pixel PNG image
+const PNG_HEADER = new Uint8Array([
+  0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0x00, 0x00, 0x00, 0x0d, 0x49,
+  0x48, 0x44, 0x52, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x08, 0x06,
+  0x00, 0x00, 0x00, 0x1f, 0x15, 0xc4, 0x89, 0x00, 0x00, 0x00, 0x0a, 0x49, 0x44,
+  0x41, 0x54, 0x08, 0xd7, 0x63, 0x60, 0x00, 0x00, 0x00, 0x02, 0x00, 0x01, 0xe2,
+  0x21, 0xbc, 0x33, 0x00, 0x00, 0x00, 0x00, 0x49, 0x45, 0x4e, 0x44, 0xae, 0x42,
+  0x60, 0x82,
+]);
+
+describe('parseFormData', () => {
+  it('should parse text fields correctly', async () => {
+    const boundary = 'boundary123';
+    const body = `--${boundary}\r\nContent-Disposition: form-data; name="field1"\r\n\r\nvalue1\r\n--${boundary}--`;
+    const contentType = `multipart/form-data; boundary=${boundary}`;
+
+    const formData = await parseFormData(
+      new TextEncoder().encode(body),
+      contentType,
+    );
+
+    expect(formData.get('field1')).toBe('value1');
+  });
+
+  it('should parse multiple text fields', async () => {
+    const boundary = 'boundary123';
+    const body = `--${boundary}\r\nContent-Disposition: form-data; name="field1"\r\n\r\nvalue1\r\n--${boundary}\r\nContent-Disposition: form-data; name="field2"\r\n\r\nvalue2\r\n--${boundary}--`;
+    const contentType = `multipart/form-data; boundary=${boundary}`;
+
+    const formData = await parseFormData(
+      new TextEncoder().encode(body),
+      contentType,
+    );
+
+    expect(formData.get('field1')).toBe('value1');
+    expect(formData.get('field2')).toBe('value2');
+  });
+
+  it('should parse file fields correctly', async () => {
+    const boundary = 'boundary123';
+    const body = `--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="test.txt"\r\nContent-Type: text/plain\r\n\r\nfile content\r\n--${boundary}--`;
+    const contentType = `multipart/form-data; boundary=${boundary}`;
+
+    const formData = await parseFormData(
+      new TextEncoder().encode(body),
+      contentType,
+    );
+
+    const file = formData.get('file') as File;
+    expect(file.name).toBe('test.txt');
+    expect(file.type).toBe('text/plain');
+    expect(file.size).toBe(12);
+    expect(await file.text()).toBe('file content');
+  });
+
+  it('should handle mixed text and file fields', async () => {
+    const boundary = 'boundary123';
+    const body = `--${boundary}\r\nContent-Disposition: form-data; name="field1"\r\n\r\nvalue1\r\n--${boundary}\r\nContent-Disposition: form-data; name="file"; filename="test.txt"\r\nContent-Type: text/plain\r\n\r\nfile content\r\n--${boundary}--`;
+    const contentType = `multipart/form-data; boundary=${boundary}`;
+
+    const formData = await parseFormData(
+      new TextEncoder().encode(body),
+      contentType,
+    );
+
+    expect(formData.get('field1')).toBe('value1');
+    const file = formData.get('file') as File;
+    expect(file.name).toBe('test.txt');
+    expect(file.type).toBe('text/plain');
+    expect(file.size).toBe(12);
+    expect(await file.text()).toBe('file content');
+  });
+
+  it('should handle empty fields', async () => {
+    const boundary = 'boundary123';
+    const body = `--${boundary}\r\nContent-Disposition: form-data; name="emptyField"\r\n\r\n\r\n--${boundary}--`;
+    const contentType = `multipart/form-data; boundary=${boundary}`;
+
+    const formData = await parseFormData(
+      new TextEncoder().encode(body),
+      contentType,
+    );
+
+    expect(formData.get('emptyField')).toBe('');
+  });
+
+  it('should handle fields with special characters', async () => {
+    const boundary = 'boundary123';
+    const body = `--${boundary}\r\nContent-Disposition: form-data; name="special"\r\n\r\n!@#$%^&*()\r\n--${boundary}--`;
+    const contentType = `multipart/form-data; boundary=${boundary}`;
+
+    const formData = await parseFormData(
+      new TextEncoder().encode(body),
+      contentType,
+    );
+
+    expect(formData.get('special')).toBe('!@#$%^&*()');
+  });
+
+  it('should handle fields with line breaks', async () => {
+    const boundary = 'boundary123';
+    const body = `--${boundary}\r\nContent-Disposition: form-data; name="multiline"\r\n\r\nLine 1\r\nLine 2\r\nLine 3\r\n--${boundary}--`;
+    const contentType = `multipart/form-data; boundary=${boundary}`;
+
+    const formData = await parseFormData(
+      new TextEncoder().encode(body),
+      contentType,
+    );
+
+    expect(formData.get('multiline')).toBe('Line 1\r\nLine 2\r\nLine 3');
+  });
+
+  it('should parse PNG file fields correctly and match input', async () => {
+    const formData = new FormData();
+    formData.append(
+      'pngFile',
+      new Blob([PNG_HEADER], { type: 'image/png' }),
+      'test.png',
+    );
+
+    // Create a Request object
+    const request = new Request('http://example.com/upload', {
+      method: 'POST',
+      body: formData,
+    });
+
+    const contentType = request.headers.get('Content-Type');
+
+    // Get the body as ArrayBuffer
+    const arrayBuffer = await request.arrayBuffer();
+
+    // Convert ArrayBuffer to Uint8Array
+    const body = new Uint8Array(arrayBuffer);
+
+    // Now use your parseFormData function
+    const parsedFormData = await parseFormData(body, contentType!);
+
+    const file = parsedFormData.get('pngFile') as File;
+    expect(file.name).toBe('test.png');
+    expect(file.type).toBe('image/png');
+    expect(file.size).toBe(PNG_HEADER.length);
+
+    const fileArrayBuffer = await file.arrayBuffer();
+    const fileUint8Array = new Uint8Array(fileArrayBuffer);
+    expect(fileUint8Array).toEqual(PNG_HEADER);
+  });
+
+  it('should handle mixed text, text file, and PNG file fields and match input', async () => {
+    const formData = new FormData();
+    formData.append('textField', 'Hello, World!');
+    formData.append(
+      'textFile',
+      new Blob(['This is a text file.'], { type: 'text/plain' }),
+      'test.txt',
+    );
+    formData.append(
+      'pngFile',
+      new Blob([PNG_HEADER], { type: 'image/png' }),
+      'test.png',
+    );
+
+    // Create a Request object
+    const request = new Request('http://example.com/upload', {
+      method: 'POST',
+      body: formData,
+    });
+
+    const contentType = request.headers.get('Content-Type');
+
+    // Get the body as ArrayBuffer
+    const arrayBuffer = await request.arrayBuffer();
+
+    // Convert ArrayBuffer to Uint8Array
+    const body = new Uint8Array(arrayBuffer);
+
+    // Now use your parseFormData function
+    const parsedFormData = await parseFormData(body, contentType!);
+
+    expect(parsedFormData.get('textField')).toBe('Hello, World!');
+
+    const textFile = parsedFormData.get('textFile') as File;
+    expect(textFile.name).toBe('test.txt');
+    expect(textFile.type).toBe('text/plain');
+    const textFileText = await textFile.text();
+    expect(textFileText).toBe('This is a text file.');
+
+    const pngFile = parsedFormData.get('pngFile') as File;
+    expect(pngFile.name).toBe('test.png');
+    expect(pngFile.type).toBe('image/png');
+    expect(pngFile.size).toBe(PNG_HEADER.length);
+    const pngFileArrayBuffer = await pngFile.arrayBuffer();
+    const pngFileUint8Array = new Uint8Array(pngFileArrayBuffer);
+    expect(pngFileUint8Array).toEqual(PNG_HEADER);
+  });
+
+  it('should handle base64 encoded content', async () => {
+    const boundary = 'boundary123';
+    const originalContent = 'Hello, World!';
+    const base64Content = Buffer.from(originalContent).toString('base64');
+    const body = `--${boundary}\r\nContent-Disposition: form-data; name="base64Field"\r\nContent-Transfer-Encoding: base64\r\n\r\n${base64Content}\r\n--${boundary}--`;
+    const contentType = `multipart/form-data; boundary=${boundary}`;
+
+    const formData = await parseFormData(
+      new TextEncoder().encode(body),
+      contentType,
+    );
+
+    const decodedContent = Buffer.from(
+      formData.get('base64Field') as string,
+      'base64',
+    ).toString();
+    expect(decodedContent).toBe(originalContent);
+  });
+
+  it('should handle binary content', async () => {
+    const formData = new FormData();
+    formData.append(
+      'binaryField',
+      new Blob([PNG_HEADER], { type: 'application/octet-stream' }),
+      'test.bin',
+    );
+
+    // Create a Request object
+    const request = new Request('http://example.com/upload', {
+      method: 'POST',
+      body: formData,
+    });
+
+    const contentType = request.headers.get('Content-Type');
+
+    // Get the body as ArrayBuffer
+    const arrayBuffer = await request.arrayBuffer();
+
+    // Convert ArrayBuffer to Uint8Array
+    const body = new Uint8Array(arrayBuffer);
+
+    // Now use your parseFormData function
+    const parsedFormData = await parseFormData(body, contentType!);
+
+    const file = parsedFormData.get('binaryField') as File;
+    expect(file.name).toBe('test.bin');
+    expect(file.type).toBe('application/octet-stream');
+    expect(file.size).toBe(PNG_HEADER.length);
+
+    const fileArrayBuffer = await file.arrayBuffer();
+    const fileUint8Array = new Uint8Array(fileArrayBuffer);
+    expect(fileUint8Array).toEqual(PNG_HEADER);
+  });
+
+  it('should handle mixed encodings in a single request', async () => {
+    const plainContent = 'Plain text';
+    const base64Content = Buffer.from('Base64 encoded text').toString('base64');
+    const binaryContent = new Uint8Array(PNG_HEADER);
+
+    const formData = new FormData();
+    formData.append('plainField', plainContent);
+    formData.append('base64Field', base64Content);
+    formData.append(
+      'binaryField',
+      new Blob([binaryContent], { type: 'application/octet-stream' }),
+      'test.bin',
+    );
+
+    // Create a Request object
+    const request = new Request('http://example.com/upload', {
+      method: 'POST',
+      body: formData,
+    });
+
+    const contentType = request.headers.get('Content-Type');
+
+    // Get the body as ArrayBuffer
+    const arrayBuffer = await request.arrayBuffer();
+
+    // Convert ArrayBuffer to Uint8Array
+    const body = new Uint8Array(arrayBuffer);
+
+    // Now use your parseFormData function
+    const parsedFormData = await parseFormData(body, contentType!);
+
+    expect(parsedFormData.get('plainField')).toBe(plainContent);
+    expect(
+      Buffer.from(
+        parsedFormData.get('base64Field') as string,
+        'base64',
+      ).toString(),
+    ).toBe('Base64 encoded text');
+
+    const file = parsedFormData.get('binaryField') as File;
+    expect(file.name).toBe('test.bin');
+    expect(file.type).toBe('application/octet-stream');
+    expect(file.size).toBe(PNG_HEADER.length);
+
+    const fileArrayBuffer = await file.arrayBuffer();
+    const fileUint8Array = new Uint8Array(fileArrayBuffer);
+    expect(fileUint8Array).toEqual(PNG_HEADER);
+  });
+});
+
+describe('bufferToString', () => {
+  it('should convert an empty ArrayBuffer to an empty string', () => {
+    const buffer = new ArrayBuffer(0);
+    expect(bufferToString(buffer)).toBe('');
+  });
+
+  it('should convert a simple ASCII string', () => {
+    const text = 'Hello, World!';
+    const buffer = new TextEncoder().encode(text).buffer;
+    expect(bufferToString(buffer)).toBe(text);
+  });
+
+  it('should handle Unicode characters', () => {
+    const text = '你好，世界！😊';
+    const buffer = new TextEncoder().encode(text).buffer;
+    expect(bufferToString(buffer)).toBe(text);
+  });
+
+  it('should handle a mix of ASCII and Unicode characters', () => {
+    const text = 'Hello, 世界! 🌍';
+    const buffer = new TextEncoder().encode(text).buffer;
+    expect(bufferToString(buffer)).toBe(text);
+  });
+
+  it('should handle a large string', () => {
+    const text = 'a'.repeat(1000000); // 1 million 'a' characters
+    const buffer = new TextEncoder().encode(text).buffer;
+    expect(bufferToString(buffer)).toBe(text);
+  });
+
+  it('should handle null characters', () => {
+    const text = 'Hello\0World';
+    const buffer = new TextEncoder().encode(text).buffer;
+    expect(bufferToString(buffer)).toBe(text);
+  });
+});