felixge/node-mysql — pull request #150 by mkiki (2011-12-04 22:54:39+00:00)
Title: Fix issue with UTF8 characters being incorrectly decoded in large queries

Hello felix,

I've been experimenting with node.js recently and using your mysql library (thanks!)
However, I ran into the following issue:
I'm running a query which fetches a few objects and stores them in a cache, that I use for quick lookup. The key is a string. Sometimes, I would get a cache miss for no reason despite the corresponding record being in the database.

Looking closely at the data, I noticed that it always occurred when the key contains non ascii chars, and indeed the key value in the cache did not match the actual value in the database. For instance, I have the "côte" key value in the database, but "C??te" in the database.

So I check all my code, and connection parameters, double check the database contents. Finally, I check the node-mysql code, and I think I found the issue.
In the query.js code, when new data comes in trough the on('data') handler, the data for a single field value may not arrive in a single call, as the 'remaining' variable shows. In this case, the 'buffer' variable holds only a part of the data and cannot be converted to UTF-8 right away because the buffer limit may be right in the middle of a multi bytes character. And that's exactly what happened.

Here's a simple fix that seems to work on my machine. It may not be the most efficient way to proceed, but it just changes a few lines.

Regards,

-Alex