jenkinsci/jenkins — pull request #478 by bmoyles (2012-05-21 23:42:59+00:00)
Title: fix for deadlock condition in jenkins.model.Jenkins.getViews()

We've run into a problem whereby folks using the keyboard shortcuts plugin are inadvertently causing UI threads to deadlock and hose the UI.

Near as we can tell, here's what's happening:

- User A is using the keyboard shortcuts plugin (as an example since this has operations that will crawl all views), asks for all views to navigate to the view
- keyboard shortcuts calls jenkins.model.Jenkins.getViews()
- jenkins.model.Jenkins.getViews() is synchronized, so now we're locked
- getViews() iterates all views and checks for permission, all via hudson.model.ViewGroupMixIn.getViews(), locking each individual view along the way through the call to hudson.security.AuthorizationStrategy.hasPermission():
```java
return base.hasPermission(a,View.CONFIGURE) || !item.getItems().isEmpty();
```
The call to item.getItems() is hudson.model.ListView.getItems(), which is synchronized, hence the lock on each view for each iteration
- User B comes along and creates a brand new item for an existing view while User A is doing their thing
- hudson.models.ListView.doCreateItem() is synchronized, so the view is locked
- doCreateItem() then attempts to do an owner.save() for each item it adds, and owner is of course, hudson.model.Hudson :)

What we end up with is the POST to create the view item grabbing a lock on the view item and then trying to lock the Hudson object to save the config

The GET request from the keyboard plugin has the Hudson/Jenkins object locked but is still iterating views. It gets to this particular view and wants to call getItems() too, but that view is locked.

Then email blows up :)

Near as we could tell, the top level call to getViews() in jenkins.model.Jenkins doesn't need to be synchronized (which could be a naive assumption, but it looked right), and we believe that dropping synchronized will clear this up.

Some thread dump foo to back up my (in-)sanity:
```
Found one Java-level deadlock:
=============================
"Handling GET /job/WE-CL-secretserverclient/search/ : RequestHandlerThread[#791]":
waiting to lock monitor 0x000000005c00ccc8 (object 0x00000006033926c0, a hudson.plugins.view.dashboard.Dashboard),
which is held by "Handling POST /view/Main/createItem : RequestHandlerThread[#654]"
"Handling POST /view/Main/createItem : RequestHandlerThread[#654]":
waiting to lock monitor 0x000000005adbf728 (object 0x00000006011d55d8, a hudson.model.Hudson),
which is held by "Handling GET /job/DSC-WAPP-stinger/ : RequestHandlerThread[#590]"
"Handling GET /job/DSC-WAPP-stinger/ : RequestHandlerThread[#590]":
waiting to lock monitor 0x000000005c00ccc8 (object 0x00000006033926c0, a hudson.plugins.view.dashboard.Dashboard),
which is held by "Handling POST /view/Main/createItem : RequestHandlerThread[#654]"

**snip**
Java stack information for the threads listed above:
===================================================
"Handling GET /job/WE-CL-secretserverclient/search/ : RequestHandlerThread[#791]":
at hudson.model.ListView.getItems(ListView.java:138)
- waiting to lock <0x00000006033926c0> (a hudson.plugins.view.dashboard.Dashboard)
at hudson.model.ListView.getItems(ListView.java:58)
at hudson.model.View.makeSearchIndex(View.java:703)
at jenkins.model.Jenkins.makeSearchIndex(Jenkins.java:1719)
at hudson.model.AbstractModelObject.getSearchIndex(AbstractModelObject.java:98)
at hudson.search.Search.doIndex(Search.java:67)

"Handling POST /view/Main/createItem : RequestHandlerThread[#654]":
at jenkins.model.Jenkins.save(Jenkins.java:2506)
- waiting to lock <0x00000006011d55d8> (a hudson.model.Hudson)
at hudson.model.ListView.doCreateItem(ListView.java:202)
- locked <0x00000006033926c0> (a hudson.plugins.view.dashboard.Dashboard)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)

"Handling GET /job/DSC-WAPP-stinger/ : RequestHandlerThread[#590]":
at hudson.model.ListView.getItems(ListView.java:138)
- waiting to lock <0x00000006033926c0> (a hudson.plugins.view.dashboard.Dashboard)
at hudson.model.ListView.getItems(ListView.java:58)
at hudson.security.AuthorizationStrategy$1.hasPermission(AuthorizationStrategy.java:103)
at hudson.security.ACL.hasPermission(ACL.java:65)
at hudson.model.View.hasPermission(View.java:503)
at hudson.model.ViewGroupMixIn.getViews(ViewGroupMixIn.java:115)
at jenkins.model.Jenkins.getViews(Jenkins.java:1370)
- locked <0x00000006011d55d8> (a hudson.model.Hudson)
at org.jenkins.ci.plugins.keyboard_shortcuts.ViewUtils.getAllViews(ViewUtils.java:47)
at org.jenkins.ci.plugins.keyboard_shortcuts.ViewUtils.getAllViewsAsJsonArray(ViewUtils.java:54)
at org.jenkins.ci.plugins.keyboard_shortcuts.KeyboardShortcutsPageDecorator.getAllViewsAsJson(KeyboardShortcutsPageDecorator
.java:101)
```