mithunr/DistCpV2-0.20.203 — Issue #1: Distcp running the job locally and failing
Author: inders
Created: 2011-12-22 13:58:53+00:00
State: open

Observing the following.

1.We call distcp via main() passing all the required arguments.
2. [1] happens continuously after a fixed lag.
3. Intermittently distcp fails with the following exception.

Env used - one node hadoop grid.

252596 [sendParams-0] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to localhost/127.0.0.1:8020 from inderbir.singh sending #434
252665 [Thread-241] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0009
java.lang.ClassCastException: org.apache.hadoop.mapreduce.lib.input.FileSplit cannot be cast to org.apache.hadoop.mapred.InputSplit
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:364)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:324)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
253157 [com.inmobi.databus.distcp.RemoteCopier_uj2_ua1] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
253152 [com.inmobi.databus.distcp.RemoteCopier_uj1_ua1] INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
252673 [IPC Client (47) connection to localhost/127.0.0.1:8020 from inderbir.singh] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to localhost/127.0.0.1:8020 from inderbir.singh got value #434
253303 [com.inmobi.databus.distcp.RemoteCopier_uj2_ua1] INFO org.apache.hadoop.mapred.JobClient - Job complete: job_local_0009
253303 [com.inmobi.databus.distcp.RemoteCopier_uj2_ua1] INFO org.apache.hadoop.mapred.JobClient - Counters: 0
253303 [Thread-247] DEBUG org.apache.hadoop.ipc.RPC - Call: addBlock 707
253304 [Thread-247] DEBUG org.apache.hadoop.hdfs.DFSClient - pipeline = 127.0.0.1:50010
253304 [Thread-247] DEBUG org.apache.hadoop.hdfs.DFSClient - Connecting to 127.0.0.1:50010
253304 [com.inmobi.databus.distcp.RemoteCopier_uj2_ua1] ERROR org.apache.hadoop.tools.DistCp - Exception encountered
java.io.IOException: DistCp failure: Job job_local_0009 has failed.
at org.apache.hadoop.tools.DistCp.execute(DistCp.java:151)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:365)
at com.inmobi.databus.distcp.RemoteCopier.fetch(RemoteCopier.java:69)
at com.inmobi.databus.AbstractCopier.run(AbstractCopier.java:61)
at java.lang.Thread.run(Thread.java:680)
253304 [Thread-247] DEBUG org.apache.hadoop.hdfs.DFSClient - Send buf size 131072
253315 [Thread-2] DEBUG org.apache.hadoop.fs.FileSystem - Removing filesystem for hdfs://gs1104.grid.corp.inmobi.com
253315 [Thread-2] DEBUG org.apache.hadoop.hdfs.DFSClient - Wait for lease checker to terminate
253315 [LeaseChecker] DEBUG org.apache.hadoop.hdfs.DFSClient - LeaseChecker is interrupted.
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
at java.lang.Thread.run(Thread.java:680)
253315 [Thread-2] DEBUG org.apache.hadoop.fs.FileSystem - Removing filesystem for hdfs://gs1104.grid.corp.inmobi.com
253316 [Thread-2] DEBUG org.apache.hadoop.fs.FileSystem - Removing filesystem for file:///
253316 [Thread-2] DEBUG org.apache.hadoop.fs.FileSystem - Removing filesystem for file:///
253316 [Thread-2] DEBUG org.apache.hadoop.fs.FileSystem - Removing filesystem for hdfs://gs1103.grid.corp.inmobi.com
253316 [Thread-2] DEBUG org.apache.hadoop.hdfs.DFSClient - Wait for lease checker to terminate
253316 [LeaseChecker] DEBUG org.apache.hadoop.hdfs.DFSClient - LeaseChecker is interrupted.
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.run(DFSClient.java:1167)
at java.lang.Thread.run(Thread.java:680)
253316 [Thread-2] DEBUG org.apache.hadoop.fs.FileSystem - Removing filesystem for hdfs://gs1103.grid.corp.inmobi.com
253489 [DataStreamer for file /databus/system/tmp/distcp-uj12011-12-22-19-0/.distcp.tmp.attempt_local_0008_m_000000_0 block blk_7354699727149582483_57471] DEBUG org.apache.hadoop.hdfs.DFSClient - DataStreamer block blk_7354699727149582483_57471 wrote packet seqno:0 size:8983 offsetInBlock:0 lastPacketInBlock:true
254005 [ResponseProcessor for block blk_7354699727149582483_57471] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient Replies for seqno 0 are SUCCESS
254005 [DataStreamer for file /databus/system/tmp/distcp-uj12011-12-22-19-0/.distcp.tmp.attempt_local_0008_m_000000_0 block blk_7354699727149582483_57471] DEBUG org.apache.hadoop.hdfs.DFSClient - Closing old block blk_7354699727149582483_57471
254006 [sendParams-0] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to localhost/127.0.0.1:8020 from inderbir.singh sending #435
254271 [IPC Client (47) connection to localhost/127.0.0.1:8020 from inderbir.singh] DEBUG org.apache.hadoop.ipc.Client - IPC Client (47) connection to localhost/127.0.0.1:8020 from inderbir.singh got value #435
254271 [Thread-244] DEBUG org.apache.hadoop.ipc.RPC - Call: complete 265
254272 [Thread-2] DEBUG org.apache.hadoop.fs.FileSystem - Removing filesystem for hdfs://localhost
254272 [Thread-244] DEBUG org.apache.hadoop.fs.FileSystem - Creating filesystem for hdfs://gs1104.grid.corp.inmobi.com/databus/streams/metric_billing/2011/12/22/13/30/gs1104.grid.corp.inmobi.com-metric_billing-2011-12-22-13-31_00000.gz
Exception in thread "Thread-2" java.lang.NullPointerException
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeThreads(DFSClient.java:3510)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3553)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3488)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.close(DFSClient.java:1133)
at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:243)
at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:349)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:1616)
at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:1586)