Project: springxd
Issue XD-3728: batch-hive module from spring-xd-samples project is not working
***Version	Spring XD Version : spring-xd-1.3.0.RELEASE, spring-xd-1.3.0.RELEASE-yarn	OS & Version: Linux 2.6.32-431.29.2.el6.x86_64	Java Version: java version "1.7.0_65" ***Description	The batch-hive job in the spring-xd-samples project (link: https://github.com/spring-projects/spring-xd-samples/tree/master/batch-hive ) is failing with inline error message. ***Steps to recreate the problem 1. Created a jar for batch-hive job.	2. Once the final jar was ready, uploaded using "module upload --name test_hive_module --type job --file /home/user/jar/batch_hive.jar" 3. After that created and deployed job using "job create --name test_hive_job --definition test_hive_module --deploy"	4. Finally launched using "job launch test_hive_job" which failed with inline error. ***Describe XD Deployment : Distributed Deployment Type : Distributed - YARN ( on AWS EC2 cloud ) Number of xd-admin’s and xd-container’s : 1 Admin and 3 Containers ***Describe Other Components Transport: Redis 3.0.1 ZooKeeper: Version 3.4.6.2.3.2.0-2950 Hadoop deployment Data Platform : Hortonworks HDP 2.3.2.0-2950 RDBMS: MySQL ***Error Message: ***************************************************** INFO DeploymentSupervisor-0 zk.ZKJobDeploymentHandler - Deployment status for job 'test_hive_job': DeploymentStatus{state=failed,error(s)=org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hive': Cannot resolve reference to bean 'hive-tasklet' while setting bean property 'tasklet'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hive-tasklet': Initialization of bean failed; nested exception is java.lang.NoClassDefFoundError: org/apache/hadoop/hive/service/HiveClient at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1481) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1226) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:305) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:301) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:196) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:753) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:835) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:537) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:686) at org.springframework.boot.SpringApplication.run(SpringApplication.java:320) at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:139) at org.springframework.xd.module.core.SimpleModule.initialize(SimpleModule.java:213) at org.springframework.xd.dirt.module.ModuleDeployer.doDeploy(ModuleDeployer.java:217) at org.springframework.xd.dirt.module.ModuleDeployer.deploy(ModuleDeployer.java:200) at org.springframework.xd.dirt.server.container.DeploymentListener.deployModule(DeploymentListener.java:365) at org.springframework.xd.dirt.server.container.DeploymentListener.deployJobModule(DeploymentListener.java:291) at org.springframework.xd.dirt.server.container.DeploymentListener.onChildAdded(DeploymentListener.java:181) at org.springframework.xd.dirt.server.container.DeploymentListener.childEvent(DeploymentListener.java:149) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:509) at org.apache.curator.framework.recipes.cache.PathChildrenCache$5.apply(PathChildrenCache.java:503) at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:92) at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:83) at org.apache.curator.framework.recipes.cache.PathChildrenCache.callListeners(PathChildrenCache.java:500) at org.apache.curator.framework.recipes.cache.EventOperation.invoke(EventOperation.java:35) at org.apache.curator.framework.recipes.cache.PathChildrenCache$10.run(PathChildrenCache.java:762) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'hive-tasklet': Initialization of bean failed; nested exception is java.lang.NoClassDefFoundError: org/apache/hadoop/hive/service/HiveClient at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:305) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:301) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:196) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) ... 37 more Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/service/HiveClient at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.getDeclaredMethods(Class.java:1975) at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:606) at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:518) at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:504) at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.determineCandidateConstructors(AutowiredAnnotationBeanPostProcessor.java:241) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.determineConstructorsFromBeanPostProcessors(AbstractAutowireCapableBeanFactory.java:1069) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1042) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:510) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:482) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:305) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:301) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:196) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:351) at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:108) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1481) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1226) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:543) ... 43 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.service.HiveClient at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 63 more } *****************************************************
Story points: 5