Issue 1:
ggn opened SPR-9190 and commented

We have been facing numerous threads being blocked waiting for getSingleton method on DefaultSingletonBeanRegistry class and because of this our servers have been hanging which requires a restart whenever it happens.

We have been collecting threaddumps whenever this happens and we are consistently seeing as below for hundreds of threads:

"catalina-exec-567" daemon prio=10 tid=0x00002aaabc5eb000 nid=0xc54 waiting for monitor entry [0x00002aaaf6320000]

java.lang.Thread.State: BLOCKED (on object monitor)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:180)

waiting to lock <0x00000005a2a58f70> (a java.util.concurrent.ConcurrentHashMap)

at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:466)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:319)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:298)

at org.springframework.context.support.AbstractApplicationContext.getBeanNamesForType(AbstractApplicationContext.java:1132)

java.lang.Thread.State: BLOCKED (on object monitor)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:180)

waiting to lock <0x00000005a2a58f70> (a java.util.concurrent.ConcurrentHashMap)

at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:466)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:319)

at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:185)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:833)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:790)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:707)

at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:433)

at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:411)

at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:543)

at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:147)

at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:84)

at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:298)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1074)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:312)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)

at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1087)

I have already commented requesting for help on the related problem https://jira.springsource.org/browse/SPR-7949 but haven't gotten any response.

https://jira.springsource.org/browse/SPR-7949 has some information about some patch but i haven't seen the patch being applied to 3.0.6 or 3.0.7 or 3.1.

Please help with this issue as this is causing some major issue with our application.

Thanks for help.

ggn

Affects: 3.0.6

Reference URL: https://jira.springsource.org/browse/SPR-7949

Issue Links:

#12604 Improve performance of DefaultSingletonBeanRegistry.getSingletonNames(..) for high-concurrent-load applications ("duplicates")

Issue 2:
mck opened SPR-7949 and commented

Under high-concurrent load situation, in an application using many prototype beans, contention occurs in DefaultSingletonBeanRegistry.getSingletonNames(..)

This method has uses singletonObjects as a mutex for all map operations.

Using a ReadWriteLock this contention could be removed, as in our case 99% of the calls here are read-only.

We see hundreds of thread dumps like in a single jvm:

java.lang.Thread.State: BLOCKED (on object monitor)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:180)

waiting to lock <0x00007fb8519b11b0> (a java.util.concurrent.ConcurrentHashMap)

at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:452)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:317)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:296)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:250)

at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1083)

java.lang.Thread.State: BLOCKED (on object monitor)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingletonNames(DefaultSingletonBeanRegistry.java:277)

waiting to lock <0x00007fb856ab64d8> (a java.util.concurrent.ConcurrentHashMap)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:352)

at org.springframework.beans.factory.BeanFactoryUtils.beanNamesForTypeIncludingAncestors(BeanFactoryUtils.java:185)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:829)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:744)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:703)

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.resolvedCachedArgument(AutowiredAnnotationBeanPostProcessor.java:436)

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.access$0(AutowiredAnnotationBeanPostProcessor.java:432)

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:469)

at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:84)

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:283)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1074)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireBean(AbstractAutowireCapableBeanFactory.java:295)

java.lang.Thread.State: BLOCKED (on object monitor)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:180)

waiting to lock <0x00007fb8519b11b0> (a java.util.concurrent.ConcurrentHashMap)

at org.springframework.beans.factory.support.AbstractBeanFactory.isTypeMatch(AbstractBeanFactory.java:452)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:317)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanNamesForType(DefaultListableBeanFactory.java:296)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBean(DefaultListableBeanFactory.java:250)

at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1083)

Affects: 3.0.4, 3.0.5

Attachments:

SPR-7949.patch (15.19 kB)

Issue Links:

#11536 Cache by-type lookups in DefaultListableBeanFactory ("duplicates")

#13828 Threads blocking on getSingleton method on DefaultSingletonBeanRegistry class ("is duplicated by")

17 votes, 16 watchers