jenkinsci/jenkins — pull request #1488 by dspartz (2014-12-16 19:24:13+00:00)
Title: Obtain hudson.model.Hudson lock prior to ListView lock to prevent

deadlock.

How to reproduce deadlock steps.
Add the following to jenkins/core/src/main/java/hudson/model/View.java line 945-946
from:

public final synchronized void doConfigSubmit( StaplerRequest req, StaplerResponse rsp ) throws IOException, ServletException, FormException {
checkPermission(CONFIGURE);

to
public final synchronized void doConfigSubmit( StaplerRequest req, StaplerResponse rsp ) throws IOException, ServletException, FormException {
try {
Thread.sleep(20000);
} catch (Exception e) {
}
checkPermission(CONFIGURE);

Launch new code
Create two jobs and one list view.
Edit view, make some change (like add a job to list). Click "OK". Thread sleep should kick in.
Within the 20 second sleep, edit a job, change name to "rename" job. Click "Save", at confirmation, click "Yes".

At the end of the 20 seconds, the deadlock should occur. Take a thread dump, like jstack. Look for "Handling POST" in the dump and you will find:

"Handling POST /jenkins/job//DoRename" thread to be "BLOCKED", waiting for lock of hudson.model.ListView. In the same stack trace, you will see that this thread has obtained a lock for hudson.model.Hudson, and hudsonmodel.FreeStyleProject.

Search again and you will find:

"Handling POST /jenkins/view/job_view/configSubmit" thread to be "BLOCKED", waiting for lock hudson.model.ListView, the same one that the prior thread has lock on.

This is the deadlock. Jenkins is now unresponsive.

See my patch for my solution (to View.java doConfigSubmit)

"Handling POST /jenkins/view/job_view/configSubmit from 10.73.213.51 : qtp1270855946-88" #88 prio=5 os_prio=0 tid=0x000000000507c800 nid=0x9355 waiting for monitor entry [0x00007fc19c2c9000]
java.lang.Thread.State: BLOCKED (on object monitor)
at jenkins.model.Jenkins.save(Jenkins.java:2756)
- waiting to lock <0x00007fc22fb40c48> (a hudson.model.Hudson)
at hudson.model.View.save(View.java:333)
at hudson.model.View.doConfigSubmit(View.java:959)
- locked <0x00007fc22ff0a4f8> (a hudson.model.ListView)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

"Handling POST /jenkins/job/job_c_11/doRename from 10.73.218.252 : qtp1270855946-93" #93 prio=5 os_prio=0 tid=0x000000000583b000 nid=0x935a waiting for monitor entry [0x00007fc19bdc4000]
java.lang.Thread.State: BLOCKED (on object monitor)
at hudson.model.ListView$Listener.renameViewItem(ListView.java:464)
- waiting to lock <0x00007fc22ff0a4f8> (a hudson.model.ListView)
at hudson.model.ListView$Listener.locationChanged(ListView.java:446)
at hudson.model.ListView$Listener.access$000(ListView.java:434)
at hudson.model.ListView$Listener$1.run(ListView.java:438)
at hudson.security.ACL.impersonate(ACL.java:145)
at hudson.model.ListView$Listener.onLocationChanged(ListView.java:436)
at hudson.model.listeners.ItemListener$6.apply(ItemListener.java:237)
at hudson.model.listeners.ItemListener$6.apply(ItemListener.java:235)
at hudson.model.listeners.ItemListener.forAll(ItemListener.java:166)
at hudson.model.listeners.ItemListener.fireLocationChange(ItemListener.java:235)
at hudson.model.AbstractItem.renameTo(AbstractItem.java:327)
- locked <0x00007fc230aa3618> (a hudson.model.FreeStyleProject)
- locked <0x00007fc22fb40c48> (a hudson.model.Hudson)
at hudson.model.Job.renameTo(Job.java:623)