Project: talendesb
Issue TESB-7061: TESB instances may hang during dependency resolution
We have multiple TESB instances in ServiceLocator and SAM tests that frequently fail to create the spring application context and thus turn to be unresponsive after one or more restarts. * This problem happens with TESB 5.1.2 and with 5.2.0! * The instances are stopped using the stop script provided with the TESB, not by killing the process. * On the level of ServiceLocator tests we see this problem frequently (on most (all?) OS platform configurations used for the 5.1.2 release tests) ** with the infrastructure TESB that contains the serviceLocatorService (and no services created / provided by QA system tests. ** with a provider instance that contains a service created by QA (including parts of the QA framework) * We have seen such problems (RARELY) also with the provider used in the ESB test suite (which only consists of a standalone ActiveMQ, a consumer and a provider TESB) On the level of the SL infrastructure instance the problem burns down to: {quote} 17:34:00,299 | INFO | ExtenderThread-9 | BundleApplicationContextListener 45 | 91 - org.springframework.osgi.extender - 1.2.1 | Application context successfully refreshed (OsgiBundleXmlApplicationContext(bundle=locator, config=osgibundle:/META-INF/spring/*.xml)) 17:34:00,308 | INFO | ExtenderThread-8 | ory.ReflectionServiceFactoryBean 441 | 112 - org.apache.cxf.cxf-rt-core - 2.6.2 | Creating Service \{http://www.talend.org/esb/sam/MonitoringService/v1}MonitoringServiceService from class org.talend.esb.sam.monitoringservice.v1.MonitoringService 17:38:58,979 | WARN | Timer-0 | WaiterApplicationContextExecutor 445 | 91 - org.springframework.osgi.extender - 1.2.1 | Timeout occurred before finding service dependencies for [OsgiBundleXmlApplicationContext(bundle=org.talend.esb.job.controller, config=osgibundle:/META-INF/spring/*.xml)] 17:38:58,980 | INFO | Timer-0 | ort.DefaultSingletonBeanRegistry 422 | 84 - org.springframework.beans - 3.0.7.RELEASE | Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@2d41717f: defining beans [cxf,org.apache.cxf.bus.spring.BusWiringBeanFactoryPostProcessor,org.apache.cxf.bus.spring.Jsr250BeanPostProcessor,org.apache.cxf.bus.spring.BusExtensionPostProcessor,org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,executorService,eventFeature,jobLauncher,jobTracker,endpointRegistry,policyProvider,providerFactory,org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean#0]; root of factory hierarchy {color:red} 17:38:58,982 | ERROR | Timer-0 | WaiterApplicationContextExecutor 424 | 91 - org.springframework.osgi.extender - 1.2.1 | Unable to create application context for [org.talend.esb.job.controller], unsatisfied dependencies: Dependency on [(objectClass=org.talend.esb.sam.agent.feature.EventFeature)] (from bean [&eventFeature]) org.springframework.context.ApplicationContextException: Application context initialization for 'org.talend.esb.job.controller' has timed out{color} org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:454)[91:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:50)[91:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:105)[91:org.springframework.osgi.extender:1.2.1] java.util.TimerThread.mainLoop(Timer.java:512)[:1.6.0_29] java.util.TimerThread.run(Timer.java:462)[:1.6.0_29] 17:38:58,983 | ERROR | Timer-0 | BundleApplicationContextListener 50 | 91 - org.springframework.osgi.extender - 1.2.1 | Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=org.talend.esb.job.controller, config=osgibundle:/META-INF/spring/*.xml)) org.springframework.context.ApplicationContextException: Application context initialization for 'org.talend.esb.job.controller' has timed out org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:454)[91:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:50)[91:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:105)[91:org.springframework.osgi.extender:1.2.1] java.util.TimerThread.mainLoop(Timer.java:512)[:1.6.0_29] java.util.TimerThread.run(Timer.java:462)[:1.6.0_29] 17:43:58,219 | WARN | JobServerCleaner | .utils.files.AbstractDataCleaner 84 | 185 - org.talend.utils - 5.1.2 | pathDir is not a directory: /opt/TESB-QA-Workspace/Sandbox/infrastructure/locatorService/container/./TalendJobServersFiles/archiveJobs 17:53:58,220 | WARN | JobServerCleaner | .utils.files.AbstractDataCleaner 84 | 185 - org.talend.utils - 5.1.2 | pathDir is not a directory: /opt/TESB-QA-Workspace/Sandbox/infrastructure/locatorService/container/./TalendJobServersFiles/archiveJobs {quote} For SAM restarting results in two spring application context timeouts: {code} 12:15:54,626 | INFO | ExtenderThread-2 | WaiterApplicationContextExecutor 243 | 95 - org.springframework.osgi.extender - 1.2.1 | No outstanding OSGi service dependencies, completing initialization for OsgiBundleXmlApplicationContext(bundle=sam-agent, config=osgibundle:/META-INF/spring/*.xml) 12:15:54,649 | INFO | ExtenderThread-4 | tory.xml.XmlBeanDefinitionReader 315 | 88 - org.springframework.beans - 3.0.7.RELEASE | Loading XML bean definitions from URL [bundleentry://220.fwk185070685/config/jmx.xml] 12:15:54,707 | INFO | ExtenderThread-5 | ent.ExecutorConfigurationSupport 115 | 90 - org.springframework.context - 3.0.7.RELEASE | Initializing ExecutorService 12:15:54,707 | INFO | ExtenderThread-5 | Context$BeanPostProcessorChecker 122 | 90 - org.springframework.context - 3.0.7.RELEASE | Bean 'defaultExecutor' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 12:15:54,707 | INFO | ExtenderThread-5 | Context$BeanPostProcessorChecker 122 | 90 - org.springframework.context - 3.0.7.RELEASE | Bean 'defaultExecutor' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 12:15:54,745 | INFO | ExtenderThread-5 | ent.ExecutorConfigurationSupport 115 | 90 - org.springframework.context - 3.0.7.RELEASE | Initializing ExecutorService 'defaultScheduler' 12:15:54,745 | INFO | ExtenderThread-5 | Context$BeanPostProcessorChecker 122 | 90 - org.springframework.context - 3.0.7.RELEASE | Bean 'defaultScheduler' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 12:15:54,754 | INFO | ExtenderThread-5 | pport.DefaultListableBeanFactory 557 | 88 - org.springframework.beans - 3.0.7.RELEASE | Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@5ede0ae6: defining beans [cxf,org.apache.cxf.bus.spring.BusWiringBeanFactoryPostProcessor,org.apache.cxf.bus.spring.Jsr250BeanPostProcessor,org.apache.cxf.bus.spring.BusExtensionPostProcessor,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,sam-agent-props,org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,samQueue,eventFeature,eventCollector,org.springframework.scheduling.annotation.internalAsyncAnnotationProcessor,org.springframework.scheduling.annotation.internalScheduledAnnotationProcessor,defaultScheduler,defaultExecutor,monitoringServiceV1Client.proxyFactory,monitoringServiceV1Client,monitoringServceV1Wrapper,clientListener,serviceListener,org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean#0]; root of factory hierarchy 12:15:54,885 | INFO | ExtenderThread-4 | WaiterApplicationContextExecutor 243 | 95 - org.springframework.osgi.extender - 1.2.1 | No outstanding OSGi service dependencies, completing initialization for OsgiBundleXmlApplicationContext(bundle=tesb.test.participants.osgi.sam.provider, config=config/osgi.xml) 12:15:54,919 | INFO | ExtenderThread-6 | ent.ExecutorConfigurationSupport 115 | 90 - org.springframework.context - 3.0.7.RELEASE | Initializing ExecutorService 12:15:54,919 | INFO | ExtenderThread-6 | Context$BeanPostProcessorChecker 122 | 90 - org.springframework.context - 3.0.7.RELEASE | Bean 'defaultExecutor' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 12:15:54,920 | INFO | ExtenderThread-6 | Context$BeanPostProcessorChecker 122 | 90 - org.springframework.context - 3.0.7.RELEASE | Bean 'defaultExecutor' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 12:15:54,932 | INFO | ExtenderThread-6 | ent.ExecutorConfigurationSupport 115 | 90 - org.springframework.context - 3.0.7.RELEASE | Initializing ExecutorService 'defaultScheduler' 12:15:54,933 | INFO | ExtenderThread-6 | Context$BeanPostProcessorChecker 122 | 90 - org.springframework.context - 3.0.7.RELEASE | Bean 'defaultScheduler' is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 12:15:54,936 | INFO | ExtenderThread-6 | pport.DefaultListableBeanFactory 557 | 88 - org.springframework.beans - 3.0.7.RELEASE | Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@15eae6fb: defining beans [cxf,org.apache.cxf.bus.spring.BusWiringBeanFactoryPostProcessor,org.apache.cxf.bus.spring.Jsr250BeanPostProcessor,org.apache.cxf.bus.spring.BusExtensionPostProcessor,org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,sam-agent-props,org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,samQueue,eventFeature,eventCollector,org.springframework.scheduling.annotation.internalAsyncAnnotationProcessor,org.springframework.scheduling.annotation.internalScheduledAnnotationProcessor,defaultScheduler,defaultExecutor,monitoringServiceV1Client.proxyFactory,monitoringServiceV1Client,monitoringServceV1Wrapper,clientListener,serviceListener,cm_sam.provider.properties,propertyPlaceholder,groovyDelegateFactory,samTestServiceProviderBean,staticProperties,SamTestServiceProviderEndpoint,NoSamTestServiceProviderEndpoint,samTestServiceProviderJmsEndpoint,SamStandardTestServiceClient4SamNoMessageContentProviderNested.proxyFactory,SamStandardTestServiceClient4SamNoMessageContentProviderNested,SamStandardTestServiceJmsNoMsgContentClientNested.proxyFactory,SamStandardTestServiceJmsNoMsgContentClientNested,jmsConfigNoMsgContent,jmsConfig,jmsConnectionFactory,methodBasedAssembler,registry,serverConnector,exporter]; root of factory hierarchy 12:15:55,553 | INFO | ExtenderThread-5 | ory.ReflectionServiceFactoryBean 441 | 116 - org.apache.cxf.cxf-rt-core - 2.7.0.SNAPSHOT | Creating Service {http://www.talend.org/esb/sam/MonitoringService/v1}MonitoringServiceService from class org.talend.esb.sam.monitoringservice.v1.MonitoringService 12:15:55,553 | INFO | ExtenderThread-6 | ory.ReflectionServiceFactoryBean 441 | 116 - org.apache.cxf.cxf-rt-core - 2.7.0.SNAPSHOT | Creating Service {http://www.talend.org/esb/sam/MonitoringService/v1}MonitoringServiceService from class org.talend.esb.sam.monitoringservice.v1.MonitoringService 12:16:55,322 | WARN | qtp534431909-46 | nsport.servlet.ServletController 156 | 123 - org.apache.cxf.cxf-rt-transports-http - 2.7.0.SNAPSHOT | Can't find the the request for http://provider:8085/services/tesb.test.participants.sam.provider/soap/StandardTestServiceProvider/'s Observer 12:20:54,482 | WARN | Timer-0 | WaiterApplicationContextExecutor 445 | 95 - org.springframework.osgi.extender - 1.2.1 | Timeout occurred before finding service dependencies for [OsgiBundleXmlApplicationContext(bundle=org.talend.esb.job.controller, config=osgibundle:/META-INF/spring/*.xml)] 12:20:54,501 | INFO | Timer-0 | ort.DefaultSingletonBeanRegistry 422 | 88 - org.springframework.beans - 3.0.7.RELEASE | Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1b814a5: defining beans [cxf,org.apache.cxf.bus.spring.BusWiringBeanFactoryPostProcessor,org.apache.cxf.bus.spring.Jsr250BeanPostProcessor,org.apache.cxf.bus.spring.BusExtensionPostProcessor,org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,executorService,eventFeature,jobLauncher,jobTracker,endpointRegistry,policyProvider,providerFactory,org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean#0]; root of factory hierarchy 12:20:54,504 | ERROR | Timer-0 | WaiterApplicationContextExecutor 424 | 95 - org.springframework.osgi.extender - 1.2.1 | Unable to create application context for [org.talend.esb.job.controller], unsatisfied dependencies: Dependency on [(objectClass=org.talend.esb.sam.agent.feature.EventFeature)] (from bean [&eventFeature]) org.springframework.context.ApplicationContextException: Application context initialization for 'org.talend.esb.job.controller' has timed out org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:454)[95:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:50)[95:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:105)[95:org.springframework.osgi.extender:1.2.1] java.util.TimerThread.mainLoop(Timer.java:512)[:1.6.0_35] java.util.TimerThread.run(Timer.java:462)[:1.6.0_35] 12:20:54,505 | ERROR | Timer-0 | BundleApplicationContextListener 50 | 95 - org.springframework.osgi.extender - 1.2.1 | Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=org.talend.esb.job.controller, config=osgibundle:/META-INF/spring/*.xml)) org.springframework.context.ApplicationContextException: Application context initialization for 'org.talend.esb.job.controller' has timed out org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:454)[95:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:50)[95:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:105)[95:org.springframework.osgi.extender:1.2.1] java.util.TimerThread.mainLoop(Timer.java:512)[:1.6.0_35] java.util.TimerThread.run(Timer.java:462)[:1.6.0_35] 12:20:54,527 | WARN | Timer-0 | WaiterApplicationContextExecutor 445 | 95 - org.springframework.osgi.extender - 1.2.1 | Timeout occurred before finding service dependencies for [OsgiBundleXmlApplicationContext(bundle=locator, config=osgibundle:/META-INF/spring/*.xml)] 12:20:54,527 | INFO | Timer-0 | ort.DefaultSingletonBeanRegistry 422 | 88 - org.springframework.beans - 3.0.7.RELEASE | Destroying singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@4d28a245: defining beans [locator-server-props,org.springframework.beans.factory.config.PropertyPlaceholderConfigurer#0,serviceLocatorBackend,serviceLocator,locatorRegistrar,defaultSelectionStrategyFactory,evenDistributionSelectionStrategyFactory,randomSelectionStrategyFactory,locatorClientEnabler,locatorManager,cxf,org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean#0,org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean#1,locatorFeature,org.springframework.osgi.service.exporter.support.OsgiServiceFactoryBean#2]; root of factory hierarchy 12:20:54,528 | ERROR | Timer-0 | WaiterApplicationContextExecutor 424 | 95 - org.springframework.osgi.extender - 1.2.1 | Unable to create application context for [locator], unsatisfied dependencies: Dependency on [(objectClass=org.apache.cxf.Bus)] (from bean [&cxf]) org.springframework.context.ApplicationContextException: Application context initialization for 'locator' has timed out org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:454)[95:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:50)[95:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:105)[95:org.springframework.osgi.extender:1.2.1] java.util.TimerThread.mainLoop(Timer.java:512)[:1.6.0_35] java.util.TimerThread.run(Timer.java:462)[:1.6.0_35] 12:20:54,528 | ERROR | Timer-0 | BundleApplicationContextListener 50 | 95 - org.springframework.osgi.extender - 1.2.1 | Application context refresh failed (OsgiBundleXmlApplicationContext(bundle=locator, config=osgibundle:/META-INF/spring/*.xml)) org.springframework.context.ApplicationContextException: Application context initialization for 'locator' has timed out org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.timeout(DependencyWaiterApplicationContextExecutor.java:454)[95:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor.access$000(DependencyWaiterApplicationContextExecutor.java:50)[95:org.springframework.osgi.extender:1.2.1] org.springframework.osgi.extender.internal.dependencies.startup.DependencyWaiterApplicationContextExecutor$WatchDogTask.run(DependencyWaiterApplicationContextExecutor.java:105)[95:org.springframework.osgi.extender:1.2.1] java.util.TimerThread.mainLoop(Timer.java:512)[:1.6.0_35] java.util.TimerThread.run(Timer.java:462)[:1.6.0_35] {code} Currently only reproducible on SL/SAM test suite using the test zones of the system test. Working on minimizing the required environment, making it easier to reproduce and collecting more details of the failed start up of the spring application context.
Story points: 1