barty/D4J-Repair huggingface dataset:
File: src/main/java/org/jsoup/nodes/Element.java
Issue: Unexpected behavior in elementSiblingIndex
The documentation for elementSiblingIndex states "Get the list index of this element in its element sibling list. I.e. if this is the first element sibling, returns 0".
This would imply that if
n=myElem.elementSiblingIndex();

then
myElem.parent().children().get(n)==myElem.  

However, this is not how elementSiblingIndex behaves. What is guaranteed is that
myElem.parent().children().get(n).equals(myElem).  

For example, if both row 2 and row 5 of a table are
<tr><td>Cell1</td><td>Cell2</td></tr>

then the Element object associated with both rows will have the same elementSiblingIndex().

Buggy code:
private static <E extends Element> Integer indexInList(Element search, List<E> elements) {
    Validate.notNull(search);
    Validate.notNull(elements);

    for (int i = 0; i < elements.size(); i++) {
        E element = elements.get(i);
        if (element.equals(search))
            return i;
    }
    return null;
}

Fixed code:
private static <E extends Element> Integer indexInList(Element search, List<E> elements) {
    Validate.notNull(search);
    Validate.notNull(elements);

    for (int i = 0; i < elements.size(); i++) {
        E element = elements.get(i);
        if (element == search)
            return i;
    }
    return null;
}
File: src/main/java/org/jsoup/nodes/Node.java
Issue: Bug in Element.insertChildren()
When using org.jsoup.nodes.Element.insertChildren(int, Collection<? extends Node>) to move (more than one!) child-elements from one parent-element to the same parent, but different index then it produces wrong results.
The problem is that the first Element's 'move' leaves the siblingIndex unchanged and then the second 'move' removes a wrong element and produces some crap. Maybe calling reindexChildren() inside the loop in addChildren() fixes this.
Version 1.8.3.
Workaround: call remove() on the elements before passing them to insertChildren()
Easy Test Case:
    @Test
    public void mustCorrectlyMoveChildrenInsideOneParentElement() {

        Document doc = new Document( "" );
        Element body = doc.appendElement( "body" );
        body.appendElement( "div1" );
        body.appendElement( "div2" );
        Element div3 = body.appendElement( "div3" );
        Element div4 = body.appendElement( "div4" );

        ArrayList<Element> toMove = new ArrayList<Element>() {
            {
                add( div3 );
                add( div4 );
            }
        };

        body.insertChildren( 0, toMove );

        String result = doc.toString().replaceAll( "\\s+", "" );
        assertEquals( "<body><div3></div3><div4></div4><div1></div1><div2></div2></body>", result );

    }

Buggy code:
protected void addChildren(int index, Node... children) {
    Validate.noNullElements(children);
    ensureChildNodes();
    for (int i = children.length - 1; i >= 0; i--) {
        Node in = children[i];
        reparentChild(in);
        childNodes.add(index, in);
    }
    reindexChildren(index);
}

Fixed code:
protected void addChildren(int index, Node... children) {
    Validate.noNullElements(children);
    ensureChildNodes();
    for (int i = children.length - 1; i >= 0; i--) {
        Node in = children[i];
        reparentChild(in);
        childNodes.add(index, in);
        reindexChildren(index);
    }
}
File: src/main/java/org/jsoup/parser/CharacterReader.java
Issue: Problem in reading XML file containing Japanese tag names
Hello,
I have XML file containing Japanese tag names and values.
JSOUP is not parsing this Japanese tags.
I am using JSOUP library (version: 1.8.3).
Please help me to solve this issue.

e.g. ( XML File to reproduce problem )
<進捗推移グラフ>
<開始予定凡例名 表示状態="0" 線色="00CED1">&#9312;&#35373;&#35336; &#38283;&#22987;&#20104;&#23450;</開始予定凡例名>
</進捗推移グラフ>
//// ****  Source Code  ******
Document doc = Jsoup.parse(XMLString.toString(),"UTF-8",Parser.xmlParser());
Elements objElementCollection = doc.getAllElements();
int iElementsSize=objElementCollection.size();
for(Element objCurrent : objElementCollection)
{
String szTag=objCurrent.tagName();
    for (TextNode tnTextNode : objCurrent.textNodes()) 
    {
        String szVal=tnTextNode.text();
    }

}

Buggy code:
boolean matchesLetter() {
    if (isEmpty())
        return false;
    char c = input[pos];
    return (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z');
}

Fixed code:
boolean matchesLetter() {
    if (isEmpty())
        return false;
    char c = input[pos];
    return (c >= 'A' && c <= 'Z') || (c >= 'a' && c <= 'z') || Character.isLetter(c);
}
File: src/main/java/org/jsoup/parser/TokenQueue.java
Issue: Parse failed with org.jsoup.select.Selector$SelectorParseException when selector has unbalanced '(' or '[' or ')' or ']'
Selector I am having as following div.card-content2:has(a.subtitle[title= MySubTitle:)]) OR a.title[title=MyTitle :] ]

Buggy code:
public String chompBalanced(char open, char close) {
    int start = -1;
    int end = -1;
    int depth = 0;
    char last = 0;

    do {
        if (isEmpty()) break;
        Character c = consume();
        if (last == 0 || last != ESC) {
            if (c.equals(open)) {
                depth++;
                if (start == -1)
                    start = pos;
            }
            else if (c.equals(close))
                depth--;
        }

        if (depth > 0 && last != 0)
            end = pos; // don't include the outer match pair in the return
        last = c;
    } while (depth > 0);
    return (end >= 0) ? queue.substring(start, end) : "";
}

Fixed code:
public String chompBalanced(char open, char close) {
    int start = -1;
    int end = -1;
    int depth = 0;
    char last = 0;
    boolean inQuote = false;

    do {
        if (isEmpty()) break;
        Character c = consume();
        if (last == 0 || last != ESC) {
            if (c.equals('\'') || c.equals('"') && c != open)
                inQuote = !inQuote;
            if (inQuote)
                continue;
            if (c.equals(open)) {
                depth++;
                if (start == -1)
                    start = pos;
            }
            else if (c.equals(close))
                depth--;
        }

        if (depth > 0 && last != 0)
            end = pos; // don't include the outer match pair in the return
        last = c;
    } while (depth > 0);
    return (end >= 0) ? queue.substring(start, end) : "";
}
File: src/main/java/org/jsoup/helper/W3CDom.java
Issue: INVALID_CHARACTER_ERR when converting Document to W3C
A recent ClearQuest version has an HTML generation bug, which is ignored by both Chrome and Internet Explorer. Jsoup.parse is also successful:
org.jsoup.nodes.Document doc = Jsoup.parse("<html><head></head><body style=\"color: red\" \"></body></html>");
(Please note the single quotation mark at the end of the body start tag.)
But trying to convert this to a W3C document fails:
new W3CDom().fromJsoup(doc);
Exception in thread "main" org.w3c.dom.DOMException: INVALID_CHARACTER_ERR: An invalid or illegal XML character is specified. 
    at org.apache.xerces.dom.CoreDocumentImpl.createAttribute(Unknown Source)
    at org.apache.xerces.dom.ElementImpl.setAttribute(Unknown Source)
    at org.jsoup.helper.W3CDom$W3CBuilder.copyAttributes(W3CDom.java:124)
    at org.jsoup.helper.W3CDom$W3CBuilder.head(W3CDom.java:92)
    at org.jsoup.select.NodeTraversor.traverse(NodeTraversor.java:31)
    at org.jsoup.helper.W3CDom.convert(W3CDom.java:66)
    at org.jsoup.helper.W3CDom.fromJsoup(W3CDom.java:46)

Perhaps copyAttributes() should ignore invalid attributes, or catch exactly this error, and ignore it, or W3CDom could have flags to ignore such errors...

Buggy code:
private void copyAttributes(org.jsoup.nodes.Node source, Element el) {
    for (Attribute attribute : source.attributes()) {
        // valid xml attribute names are: ^[a-zA-Z_:][-a-zA-Z0-9_:.]
        String key = attribute.getKey().replaceAll("[^-a-zA-Z0-9_:.]", "");
            el.setAttribute(key, attribute.getValue());
    }
}

Fixed code:
private void copyAttributes(org.jsoup.nodes.Node source, Element el) {
    for (Attribute attribute : source.attributes()) {
        // valid xml attribute names are: ^[a-zA-Z_:][-a-zA-Z0-9_:.]
        String key = attribute.getKey().replaceAll("[^-a-zA-Z0-9_:.]", "");
        if (key.matches("[a-zA-Z_:]{1}[-a-zA-Z0-9_:.]*"))
            el.setAttribute(key, attribute.getValue());
    }
}
File: src/main/java/org/jsoup/parser/TokeniserState.java
Issue: Parse slash in attibutes
Hello,
I don't know if it is a bug or not, but when I'm parsing:
<img /onerror="a()"/>
The result of the parsers is:
<img nerror="a()"/>
Is it OK? can I change the parser behavior for those types of tags?

Buggy code:
void read(Tokeniser t, CharacterReader r) {
    char c = r.consume();
    switch (c) {
        case '>':
            t.tagPending.selfClosing = true;
            t.emitTagPending();
            t.transition(Data);
            break;
        case eof:
            t.eofError(this);
            t.transition(Data);
            break;
        default:
            t.error(this);
            t.transition(BeforeAttributeName);
    }
}

Fixed code:
void read(Tokeniser t, CharacterReader r) {
    char c = r.consume();
    switch (c) {
        case '>':
            t.tagPending.selfClosing = true;
            t.emitTagPending();
            t.transition(Data);
            break;
        case eof:
            t.eofError(this);
            t.transition(Data);
            break;
        default:
            t.error(this);
            r.unconsume();
            t.transition(BeforeAttributeName);
    }
}
File: src/main/java/org/jsoup/nodes/Attributes.java
Issue: removeIgnoreCase ConcurrentModificationException
When testing out the removeIgnoreCase method, I'm now seeing a ConcurrentModificationException with code like: element.select("abc").first().removeAttr("attr1").removeAttr("attr2");
It appears to be due to using a foreach loop over the LinkedHashMap to do the removal. Changing to do the removal directly with an iterator fixes this issue.
Like so:
for (Iterator<Map.Entry<String, Attribute>> iter = attributes.entrySet().iterator(); iter.hasNext();) {
            Map.Entry<String, Attribute> entry = iter.next();
            if (entry.getKey().equalsIgnoreCase("key1")) {
                iter.remove();
            }
        }

Buggy code:
public void removeIgnoreCase(String key) {
    Validate.notEmpty(key);
    if (attributes == null)
        return;
    for (Iterator<String> it = attributes.keySet().iterator(); it.hasNext(); ) {
        String attrKey = it.next();
        if (attrKey.equalsIgnoreCase(key))
            attributes.remove(attrKey);
    }
}

Fixed code:
public void removeIgnoreCase(String key) {
    Validate.notEmpty(key);
    if (attributes == null)
        return;
    for (Iterator<String> it = attributes.keySet().iterator(); it.hasNext(); ) {
        String attrKey = it.next();
        if (attrKey.equalsIgnoreCase(key))
            it.remove();
    }
}
File: src/main/java/org/jsoup/parser/Token.java
Issue: Jsoup.clean control characters throws: IllegalArgumentException: String must not be empty
I found that when running Jsoup.clean() on a string that contains the format below, Jsoup throws: IllegalArgumentException: String must not be empty.
The problematic string format:
'<a/*>', (where * is a control char).
i.e. < char followed by a letter (a-z), then any chars, / and any control char (ASCII 0-31) except 0, 9-10, 12-13, any chars, and a > char.

Buggy code:
final void newAttribute() {
    if (attributes == null)
        attributes = new Attributes();

    if (pendingAttributeName != null) {
        // the tokeniser has skipped whitespace control chars, but trimming could collapse to empty for other control codes, so verify here
        pendingAttributeName = pendingAttributeName.trim();
            Attribute attribute;
            if (hasPendingAttributeValue)
                attribute = new Attribute(pendingAttributeName,
                    pendingAttributeValue.length() > 0 ? pendingAttributeValue.toString() : pendingAttributeValueS);
            else if (hasEmptyAttributeValue)
                attribute = new Attribute(pendingAttributeName, "");
            else
                attribute = new BooleanAttribute(pendingAttributeName);
            attributes.put(attribute);
    }
    pendingAttributeName = null;
    hasEmptyAttributeValue = false;
    hasPendingAttributeValue = false;
    reset(pendingAttributeValue);
    pendingAttributeValueS = null;
}

Fixed code:
final void newAttribute() {
    if (attributes == null)
        attributes = new Attributes();

    if (pendingAttributeName != null) {
        // the tokeniser has skipped whitespace control chars, but trimming could collapse to empty for other control codes, so verify here
        pendingAttributeName = pendingAttributeName.trim();
        if (pendingAttributeName.length() > 0) {
            Attribute attribute;
            if (hasPendingAttributeValue)
                attribute = new Attribute(pendingAttributeName,
                    pendingAttributeValue.length() > 0 ? pendingAttributeValue.toString() : pendingAttributeValueS);
            else if (hasEmptyAttributeValue)
                attribute = new Attribute(pendingAttributeName, "");
            else
                attribute = new BooleanAttribute(pendingAttributeName);
            attributes.put(attribute);
        }
    }
    pendingAttributeName = null;
    hasEmptyAttributeValue = false;
    hasPendingAttributeValue = false;
    reset(pendingAttributeValue);
    pendingAttributeValueS = null;
}
File: src/main/java/org/jsoup/nodes/Entities.java
Issue: StringIndexOutOfBoundsException when parsing link http://news.yahoo.com/s/nm/20100831/bs_nm/us_gm_china
java.lang.StringIndexOutOfBoundsException: String index out of range: 1
at java.lang.String.charAt(String.java:686)
at java.util.regex.Matcher.appendReplacement(Matcher.java:711)
at org.jsoup.nodes.Entities.unescape(Entities.java:69)
at org.jsoup.nodes.TextNode.createFromEncoded(TextNode.java:95)
at org.jsoup.parser.Parser.parseTextNode(Parser.java:222)
at org.jsoup.parser.Parser.parse(Parser.java:94)
at org.jsoup.parser.Parser.parse(Parser.java:54)
at org.jsoup.Jsoup.parse(Jsoup.java:30)

Buggy code:
static String unescape(String string) {
    if (!string.contains("&"))
        return string;

    Matcher m = unescapePattern.matcher(string); // &(#(x|X)?([0-9a-fA-F]+)|[a-zA-Z]+);?
    StringBuffer accum = new StringBuffer(string.length()); // pity matcher can't use stringbuilder, avoid syncs
    // todo: replace m.appendReplacement with own impl, so StringBuilder and quoteReplacement not required

    while (m.find()) {
        int charval = -1;
        String num = m.group(3);
        if (num != null) {
            try {
                int base = m.group(2) != null ? 16 : 10; // 2 is hex indicator
                charval = Integer.valueOf(num, base);
            } catch (NumberFormatException e) {
            } // skip
        } else {
            String name = m.group(1);
            if (full.containsKey(name))
                charval = full.get(name);
        }

        if (charval != -1 || charval > 0xFFFF) { // out of range
            String c = Character.toString((char) charval);
            m.appendReplacement(accum, c);
        } else {
            m.appendReplacement(accum, m.group(0));
        }
    }
    m.appendTail(accum);
    return accum.toString();
}

Fixed code:
static String unescape(String string) {
    if (!string.contains("&"))
        return string;

    Matcher m = unescapePattern.matcher(string); // &(#(x|X)?([0-9a-fA-F]+)|[a-zA-Z]+);?
    StringBuffer accum = new StringBuffer(string.length()); // pity matcher can't use stringbuilder, avoid syncs
    // todo: replace m.appendReplacement with own impl, so StringBuilder and quoteReplacement not required

    while (m.find()) {
        int charval = -1;
        String num = m.group(3);
        if (num != null) {
            try {
                int base = m.group(2) != null ? 16 : 10; // 2 is hex indicator
                charval = Integer.valueOf(num, base);
            } catch (NumberFormatException e) {
            } // skip
        } else {
            String name = m.group(1);
            if (full.containsKey(name))
                charval = full.get(name);
        }

        if (charval != -1 || charval > 0xFFFF) { // out of range
            String c = Character.toString((char) charval);
            m.appendReplacement(accum, Matcher.quoteReplacement(c));
        } else {
            m.appendReplacement(accum, Matcher.quoteReplacement(m.group(0))); // replace with original string
        }
    }
    m.appendTail(accum);
    return accum.toString();
}
File: src/main/java/org/jsoup/parser/HtmlTreeBuilderState.java
Issue: Wrong parsing of case sensitive HTML
Executing :
        String xml="<r><X>A</X><y>B</y></r>";
	Parser parser = Parser.htmlParser();
	parser.settings(ParseSettings.preserveCase);
	org.jsoup.nodes.Document _doc = parser.parseInput(xml, "/");

Results in :
<html>
<head></head>
<body>
<r>
<X>
A
<y>
B
</y>
</X>
</r>
</body>
</html>
Manual hacking : remove all .toLowerCase() invocations from Token.java (normalName=...)

Buggy code:
boolean anyOtherEndTag(Token t, HtmlTreeBuilder tb) {
    String name = t.asEndTag().normalName();
    ArrayList<Element> stack = tb.getStack();
    for (int pos = stack.size() -1; pos >= 0; pos--) {
        Element node = stack.get(pos);
        if (node.nodeName().equals(name)) {
            tb.generateImpliedEndTags(name);
            if (!name.equals(tb.currentElement().nodeName()))
                tb.error(this);
            tb.popStackToClose(name);
            break;
        } else {
            if (tb.isSpecial(node)) {
                tb.error(this);
                return false;
            }
        }
    }
    return true;
}

Fixed code:
boolean anyOtherEndTag(Token t, HtmlTreeBuilder tb) {
    String name = t.asEndTag().name(); // matches with case sensitivity if enabled
    ArrayList<Element> stack = tb.getStack();
    for (int pos = stack.size() -1; pos >= 0; pos--) {
        Element node = stack.get(pos);
        if (node.nodeName().equals(name)) {
            tb.generateImpliedEndTags(name);
            if (!name.equals(tb.currentElement().nodeName()))
                tb.error(this);
            tb.popStackToClose(name);
            break;
        } else {
            if (tb.isSpecial(node)) {
                tb.error(this);
                return false;
            }
        }
    }
    return true;
}
File: src/main/java/org/jsoup/parser/HtmlTreeBuilderState.java
Issue: Incorrect handling of self-closing tags noframes, style and title cause remainder of document to be html-escaped
Given the input:
<html>
<head>
	<style />   <!-- < - - this is the culprit -->
</head>
<body>
	<p>Whatever</p>
</body>
</html>
JSoup 1.8.2 and also http://try.jsoup.org/~lJwWpjXYUSTBeBZhdEnS3Mt56g4 will produce:
    <html>
     <head> 
      <style></style>
     </head>
     <body>
       &lt;/head&gt; &lt;body&gt; &lt;p&gt;Whatever&lt;/p&gt; &lt;/body&gt; &lt;/html&gt;
     </body>
    </html>
With <title/> instead of <style/>, the result is different but still wrong (http://try.jsoup.org/~BZ3uoMki-r904fZxUOWJgLJO7r8 ):
<html>
 <head> 
  <title></title>
 </head>
 <body>
   &lt;/head&gt;  
  <p>Whatever</p>  
 </body>
</html>
That weirdness was fixed for <script> with Issue #305: http://try.jsoup.org/~3Ms6TQCrrdaA_uPgxgURYYvwFAg
<html>
 <head> 
  <script></script> 
 </head> 
 <body> 
  <p>Whatever</p>  
 </body>
</html>
Looking at the source, it seems only the HtmlTreeBuilderState handling for <noframes>, <style> and <title> in the methods handleRawText and handleRcData doesn't get along with the self-closing tags.
Any other tagname I've checked (and I tried to cover all branches of that case StartTag switch) results in a good parse similar to the <script> case, which is what I'd expect.
Thanks for looking into this!

Buggy code:
private static void handleRawtext(Token.StartTag startTag, HtmlTreeBuilder tb) {
    tb.insert(startTag);
    tb.tokeniser.transition(TokeniserState.Rawtext);
    tb.markInsertionMode();
    tb.transition(Text);
}

Fixed code:
private static void handleRawtext(Token.StartTag startTag, HtmlTreeBuilder tb) {
    tb.tokeniser.transition(TokeniserState.Rawtext);
    tb.markInsertionMode();
    tb.transition(Text);
    tb.insert(startTag);
}
File: src/main/java/org/jsoup/parser/HtmlTreeBuilder.java
Issue: version 1.11.1 java.lang.StackOverflowError
version 1.10.3 no problem
version 1.11.1 java.lang.StackOverflowError
Example URL：
http://szshb.nxszs.gov.cn/
http://www.lnfsfda.gov.cn/
http://www.beihai.gov.cn/
http://www.fsepb.gov.cn/
http://www.bhem.gov.cn

Buggy code:
private boolean inSpecificScope(String[] targetNames, String[] baseTypes, String[] extraTypes) {
    // https://html.spec.whatwg.org/multipage/parsing.html#has-an-element-in-the-specific-scope
    int bottom = stack.size() -1;
    if (bottom > MaxScopeSearchDepth) {
        bottom = MaxScopeSearchDepth;
    }
    final int top = bottom > MaxScopeSearchDepth ? bottom - MaxScopeSearchDepth : 0;
    // don't walk too far up the tree

    for (int pos = bottom; pos >= top; pos--) {
        final String elName = stack.get(pos).nodeName();
        if (inSorted(elName, targetNames))
            return true;
        if (inSorted(elName, baseTypes))
            return false;
        if (extraTypes != null && inSorted(elName, extraTypes))
            return false;
    }
    //Validate.fail("Should not be reachable"); // would end up false because hitting 'html' at root (basetypes)
    return false;
}

Fixed code:
private boolean inSpecificScope(String[] targetNames, String[] baseTypes, String[] extraTypes) {
    // https://html.spec.whatwg.org/multipage/parsing.html#has-an-element-in-the-specific-scope
    final int bottom = stack.size() -1;
    final int top = bottom > MaxScopeSearchDepth ? bottom - MaxScopeSearchDepth : 0;
    // don't walk too far up the tree

    for (int pos = bottom; pos >= top; pos--) {
        final String elName = stack.get(pos).nodeName();
        if (inSorted(elName, targetNames))
            return true;
        if (inSorted(elName, baseTypes))
            return false;
        if (extraTypes != null && inSorted(elName, extraTypes))
            return false;
    }
    //Validate.fail("Should not be reachable"); // would end up false because hitting 'html' at root (basetypes)
    return false;
}
File: src/main/java/org/jsoup/nodes/Element.java
Issue: Whitespaces not properly handled in <pre> tag
If a "pre" tag contains deep nested tags, whitespaces in nested tags are not preserved.
Example:
String s = "<pre><code>\n"
        + "  message <span style=\"color:red\"> other   \n    message  with \n"
        + "   whitespaces      </span>\n"
        + "</code></pre>";
    Document doc = Jsoup.parse(s);
    System.out.println(doc.select("pre").first().outerHtml());

Will output:
<pre><code>
  message <span style="color:red"> other message with whiptespaces </span>
</pre></code>

Output is OK if we omit the "code" tag

Buggy code:
static boolean preserveWhitespace(Node node) {
    // looks only at this element and five levels up, to prevent recursion & needless stack searches
    if (node != null && node instanceof Element) {
        Element el = (Element) node;
            if (el.tag.preserveWhitespace())
                return true;
            else
                return el.parent() != null && el.parent().tag.preserveWhitespace();
    }
    return false;
}

Fixed code:
static boolean preserveWhitespace(Node node) {
    // looks only at this element and five levels up, to prevent recursion & needless stack searches
    if (node != null && node instanceof Element) {
        Element el = (Element) node;
        int i = 0;
        do {
            if (el.tag.preserveWhitespace())
                return true;
            el = el.parent();
            i++;
        } while (i < 6 && el != null);
    }
    return false;
}