Package: pydistcp
pydistcpA python WebHDFS/HTTPFS based tool for inter/intra-cluster data copying. This tool is very suitable for multiple mid or small size files cross-clusters copy. Compared to the normal distcp which adds a lot of overhead time for submitting the map-reduce job then waiting for YARN to schedule it...,  pydistcp uses webhdfs to stream the data from source cluster datanodes directly to destination cluster datanodes using multiple parallel threads.When transferring few huge files, the normal distcp may be faster, but when transferring lot of small, midsize or relatively big file,  pydistcp provides a very good performance.$pydistcp-f-sstaging-dprod/data/outgoing/data/incoming--threads=10--part-size=13107227.1%[pending:32|transferring:6|complete:4]JobStatus:{"Size Failed":0,"Size Copied":257721641,"Source Path":"/data/t100","Size Expected":257721641,"Files Expected":42,"Files Failed":0,"Destination Path":"/data/t200","Start Time":"2017-02-22 17:39:29","Files Skipped":0,"Size Deleted":0,"End Time":"2017-02-22 17:39:50","Files Copied":42,"Files Deleted":0,"Duration":20.756325006484985,"Outcome":"Successful","Size Skipped":0}Pydistcp usespywhdfsfor establishing connections with WEBHDFS/HTTPFS source and destination clusters.FeaturesPydistcp is based on pywhdfs project to establish WebHDFS and HTTPFS connections with source and destination clusters,
so all clusters configurations supported in  pywhdfs are also supported in pydistcp:Support both secure (Kerberos,Token) and insecure clustersSupports HA cluster and handle namenode failoverSupports HDFS federation with multiple nameservices and mount points.Supports data copy between secure and insecure clustersSupports data copy between clusters using different kerberos realms using token authenticationSupports data copy between encrypted and unencrypted clustersJson format clusters configuration.Perform concurrent multithreaded data copy.Getting started$easy_installpydistcpConfigurationPydistcp share the same json configuration file used bypywhdfs.
Please refer to the project readme file for details about the json configuration schema.USAGEThere are multiple arguments you can use to alter the way the copy works, or to enhance the performance of the job depending on the size of the server you use.
Use the help argument to display the full list of supported parameters:$pydistcp--helppydistcp:ApythonWebHDFSbasedtoolforinter/intra-clusterdatacopying.Usage:pydistcp[-fp][--no-checksum][--silent](-sCLUSTER-dCLUSTER)[-v...][--part-size=PART_SIZE][--threads=THREADS]SRC_PATHDEST_PATHpydistcp(--version|-h)Options:--versionShowversionandexit.-h--helpShowhelpandexit.-sCLUSTER--src=CLUSTERAliasofsourcenamenodetoconnectto(validonlywithdist).-dCLUSTER--dest=CLUSTERAliasofdestinationnamenodetoconnectto(validonlywithdist).-v--verboseEnablelogoutput.Canbespecifiedmultipletimestoincreaseverbosityeachtime.--no-checksumDisablechecksumcheckpriortofiletransfer.Thiswillforceoverwrite.--silentDon'tdisplayprogressstatus.-f--forceAllowoverwritinganyexistingfiles.-p--preservePreservefileattributes.--threads=THREADSNumberofthreadstouseforparallelization.zerolimitstheconcurrencytothemaximumconcurrentthreadssupportedbythecluster.[default:0]--part-size=PART_SIZEIntervalinbytesbywhichthefileswillbecopiedneedstobeaPowersof2.[default:65536]Examples:pydistcp-sprod-dpreprod-v/tmp/src/tmp/destAll cluster connection parameters will be fetched from the json configuration file.benchmarksBelow some benchmarks showing the impact of data size on the copy performance using pydistcp :File CountData SizeTime237911.4 G4m39.069s24225.9 G5m39.348s869116.9 G25m53.231s42545.8 M0m19.946s17885.2 G2m25.649s442835.7 G10m20.129s23575.6 G3m2.598s1802.3 G0m33.133s3347.6 G1m26.260sNote that all test cases are executed with 10 concurrent threads on a machine having 6 cores and supporting up to 12 threads and no files
are skipped during the copy. Both the source and destination clusters are secured with kerberos and use ssl to encrypt transferred data.Pydistcp performance may be impact by lot of parameters like:the size of the machine performing the copy.The type of the source and destination clusters (secure clusters with kerberos does not support lot of concurrent threads, it is better from a performance perspective to use token authentication)SSL and the length of encryption key usedThe type of data to be transferred : Pydistcp deliver the best performance for multiple files having approximately uniform sizes.ContributingFeedback and Pull requests are very welcome!