Project: JDT
Bug 57903 [implementation] BufferedDocumentScanner.read is broken for partial documents
Reported: 2004-04-08 10:54:50 | Status: resolved fixed

3.0M8 Possible fix: Replace line 3 below with if (fBufferOffset + fBufferLength == fRangeOffset+fRangeLength) I am writing a colorizer for jsp pages. As part of this effort, I am using the FastJavaPartitioner to partition the scriptlet code into the correct java chunks. So the crux of the problem is that I am only partitioning part of a document. In doing so, I ran into a problem in BufferedDocumentScanner.read() that does not correctly test for end of partitioning range and ends up throwing an ArrayIndexOutOfBounds exception (in some cases, depending on the exact data). I've included my test program and the code from BufferedDocumentScanner. The problem appears to be that on line 3 the test is done against the length of the document instead of fRangeOffset+fRangeLength. This means that instead of returning EOF, updateBuffer() is called and fOffset is set to 0 and fBuffer [0] is returned without checking to make sure that the range is not ended. The next time through read, fOffset is 1, fBufferLength is 0, and garbage is returned until the end of the array is reached. This doesn't cause the exception all the time (although garbage data is still returned), because of vagaries with the FastJavaPartitioner state machine. Exception trace: java.lang.ArrayIndexOutOfBoundsException: 1000 at org.eclipse.jdt.internal.ui.text.BufferedDocumentScanner.read (BufferedDocumentScanner.java:115) at org.eclipse.jdt.internal.ui.text.FastJavaPartitionScanner.nextToken (FastJavaPartitionScanner.java:101) at FastPartitionScannerTest.main(FastPartitionScannerTest.java:34) Program demonstrating bug: import org.eclipse.jdt.internal.ui.javaeditor.PartiallySynchronizedDocument; import org.eclipse.jdt.internal.ui.text.FastJavaPartitionScanner; import org.eclipse.jface.text.rules.Token; public class FastPartitionScannerTest { public static void main(String[] args) { FastJavaPartitionScanner scanner=new FastJavaPartitionScanner (); PartiallySynchronizedDocument document=new PartiallySynchronizedDocument(); document.set("<%/**f%>"); scanner.setRange(document,2,4); Token t=(Token) scanner.nextToken(); System.out.println(t.toString()); } } BufferedDocumentScanner.read(): 1 public final int read() { 2 if (fOffset == fBufferLength) { 3*** if (fBufferOffset + fBufferLength == fDocument.getLength()) 4 return EOF; 5 else { 6 updateBuffer(fBufferOffset + fBufferLength); 7*** fOffset= 0; 8 } 9 } 10 return fBuffer[fOffset++]; 11 }

Files changed in fix:
org.eclipse.jdt.text.tests/src/org/eclipse/jdt/text/tests/PartitionTokenScannerTest.java org.eclipse.jdt.ui/ui/org/eclipse/jdt/internal/ui/text/BufferedDocumentScanner.java