Change:
VS Code version: Code - Insiders 1.48.0-insider (, 2020-07-10T11:55:11.286Z) OS version: Darwin x64 19.5.0 All extensions disabledSteps to Reproduce: Step 1. Download the file Step 2. Check line 982 of the file with some other editors Step 3. Open the file in vscode and check line 982 Expected behaviour: the line is read without any change (i.e., the content should be exactly the same as the file on disk). Actual behaviour: the line is changed by vscode. Here is the screenshot: ! <hr /<bThe following information is generated by VSCode's Report Issue</bIssue Type: <bBug</bWhen opening the given file, line 982 of the file is changed automatically with one (or two?) wrong characters. Step to reproduce: open the file and check line 982. Expected behaviour: the line is read without any change (i.e., the content should be exactly the same as the file on disk). Actual behaviour: the line is changed by vscode. Here is the screenshot: ! Here is the file (filename changed to , but it doesn't matter): VS Code version: Code - Insiders 1.48.0-insider (, 2020-07-10T11:55:11.286Z) OS version: Darwin x64 19.5.0 <details<summarySystem Info</summary </detailsExtensions: none <!-- generated by issue reporter --
I updated the post with a better format. The detailed system info is folded at the end of the post (generated by vscode's Report Issue). I'd like to help if more info is still needed.
Thanks, I can reproduce with the following steps: create an empty folder init git and add commit everything open 2 characters are not displayed properly and thus appear as change ! This is a regression from where I am no longer decoding via when the encoding is UTF-8. I might need your help here. Is there anything does for UTF-8 that would explain it? My naive assumption would be that for UTF-8, is not decoding anything and simply returns the buffer that was given, but stepping through the code with a debugger, I am seeing a few lines where I am not sure if they could be an explanation. For example:
Is this utf8 decoding? The problem could be that a utf byte sequence that corresponds to a single character is split into 2 chunks, so naive decoding of chunks doesn't work. iconv-lite uses StringDecoder class to handle situations like this.
yes, this is for UTF-8 decoding. Can you elaborate on that, trying to understand how I could resolve this in VSCode without having to load for UTF-8 at least. We read the text into 64kb chunks of . Then we call to get 64kb chunks of text to display in the editor. Is there a better way you say? Code pointer: Interestingly, this seems to be related to our 64kb chunk size. If I change our chunks to be very large (e.g. so that this file fits into a single chunk), the result of does not reproduce this bug. But of course that would not be a good solution.
I think this is happening when a multi-byte character ends up on the 64kb boundary. Let's say it is 2 byte character. Then you get 1 byte in the first chunk and 1 in the second. By simply calling on the buffers you end up with it being interpreted as 2 single byte characters and thus it gets corrupted. The solution would be to use streaming converter such as or . Notice that exists in both - browser and node, so likely is a better choice.
thanks, can I ask you to check out this change towards that direction, it seems to resolve the issue with the file at least: However I can see a test failure from one of your tests now:
I am actually not 100% sure is stateless, but we seem to be keeping a shared instance around in too. Would be good to verify.
TextDecoder is definitely not stateless afaik. It assumes decode() is being called on successive chunks.
Yeah, I will double Alex here. My understanding is also that is stateful, because it might preserve certain bytes in between calls. Going back to the test - it might be that and use a bit different algorithms of how to convert corrupted bytes to string. However this is not what my test was intended to test. My test was about checking that we are calling before ending the stream. In try to do the following before your change and you will see the error Now to make my test work as intended with your change it needs to be changed to the following Then try removing call to in again to see that it catches the mistake.
we seem to share the same instance though in our base type: Maybe as author of this could chime in, but I believe that a text encoder/decoder is not stateful, unless you use the property. In my case likely I should switch to a model where the type is created each time it is needed. I will try out the suggested changes to fix the test later, thanks.
I pushed this by always creating a new instance. Compared to what we had before, we now go through when previously we did go through . I really hope this does not lead into perf issues where is slower for some reason, but I would not expect so given we only use it for .
:+1: Yes, string decoding/encoding needs to take care that byte sequences are not broken up at chunks starts/ends. shares the same instance of /, but that is OK, because does not give any state guarantees, i.e. users of and need to handle chunks edges, just like node's .

Discussion:
* This stream will only load iconv-lite lazily if the encoding * is not UTF-8. This ensures that for most common cases we do * not pay the price of loading the module from disk. <ins> * * We still need to be careful when converting UTF-8 to a string * though because we read the file in chunks of Buffer and thus * need to decode it via TextDecoder helper that is available * in browser and node.js environments. </ins> */ static async create(encoding: string): Promise<DecoderStream> { let decoder: IDecoderStream | undefined = undefined; if (encoding !== UTF8) { const iconv = await import('iconv-lite-umd'); decoder = iconv.getDecoder(toNodeEncoding(encoding)); <ins> } else { const utf8TextDecoder = new TextDecoder(); decoder = { write(buffer: Uint8Array): string { return utf8TextDecoder.decode(buffer, { // Signal to TextDecoder that potentially more data is coming // and that we are calling `decode` in the end to consume any // remainders stream: true }); }, end(): string | undefined { return utf8TextDecoder.decode(); } }; </ins> } return new DecoderStream(decoder); } <del> private constructor(private iconvLiteDecoder: IDecoderStream | undefined) { } </del> <ins> private constructor(private iconvLiteDecoder: IDecoderStream) { } </ins> write(buffer: Uint8Array): string { if (this.iconvLiteDecoder) {

const source = newTestReadableStream(buffers); const { stream } = await encoding.toDecodeStream(source, { minBytesRequiredForDetection: 4, guessEncoding: false, overwriteEncoding: async detected => detected || encoding.UTF8 }); <del> const expected = incompleteEmojis.toString(encoding.UTF8); </del> <ins> const expected = new TextDecoder().decode(incompleteEmojis); </ins> const actual = await readAllAsString(stream); assert.equal(actual, expected);