jhy/jsoup — Issue #145: Entity whose name is made up of letters and digits is not retained
Author: helsom
Created: 2011-12-01 10:01:37+00:00
State: open

At about line 136 Tokenizer.java, reader.consumeLetterSequence() is called. This is fine until it an entity such as &sup1; is encountered - just the letter part of the entity name will be read causing the Entities.isNamedEntity(nameRef) call at about line 140 to fail.

I have fixed this quickly locally by replacing reader.consumerLetterSequence() with a call to a new consumeLetterDigitSequence() in the CharacterReader.java - there may be a better way of doing this:

String consumeLetterDigitSequence() {
String letters = consumeLetterSequence();
String digits = consumeDigitSequence();
return letters + digits;
}

The following is a sample unit test:

@Test public void letterDigitEntities() {
String html = "

¹²³¼½¾

";
Whitelist whitelist = Whitelist.none();
whitelist
.addTags("p");
String html = Jsoup.clean(html, whitelist);
assertEquals("

¹²³¼½¾

", html);
}