cloudera/cdk — pull request #3 by codingtony (2013-05-22 21:57:15+00:00)
Title: Tool that helps adding jars to HDFS and to the Distributed Cache that allow fast reuse of libs between jobs

I'm not sure I'm doing this properly, but we wanted to contribute to CDK with this addition.

**Little description of the tool**

This tool helps to setup the job classpath at runtime. It allows library sharing between job. That result in faster jobs setup (since most of the time the libs are already uploaded in HDFS).

Before submitting a job, you use this tool to provide the classes that you use in your job.

The tool will find the jar(s), or will create the jars and upload them to a "library" path in HDFS, and it will create an md5 file along the uploaded jar.

If another job needs the same jar and provide the same "library" path it will discover it and use it, without having to lose the time that the upload of the jar would require.

If the jar does not exist in the "library" path, it will upload it.
However, if the jar is already in the "library" path, the tool will compute the md5 of the jar and compare with the one found in HDFS, and if there's a difference, the jar will be uploaded.

If it creates a jar (from the classes of the job itself or from the classes in your workspace for example), it will upload the created jar to the "library" path and clean them after the JVM exits.

It makes the developers happy, since the workspace code is always automatically uploaded to HDFS properly. And more importantly, once the code is deployed in production it continues to work!

We currently use the code in our environment I adapted it a bit to make it more CDK-friendly.

I'm expecting to receive comments, I'm also aware that the documentation, test and style are most likely not adequate, but I wanted to release it early to get feedback faster.

I will be glad to fix everything that might come up.

**Snippet**

Sample of a job driver that would require HashFunction from Guava

```java
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
import org.apache.log4j.BasicConfigurator;

import com.google.common.hash.HashFunction;

public class TestTool extends Configured implements Tool {

/**
* @param args
*/
public static void main(String[] args) throws Exception{
BasicConfigurator.configure();
ToolRunner.run(new TestTool(), args);
}

@Override
public int run(String[] args) throws Exception {

// Could be a static call, or I could use the Builder pattern style
JobClasspathHelper jch = new JobClasspathHelper();

// be sure you have write access to that path
// We want to upload the class
jch.prepareClasspath(getConf(), new Path("/lib/path"), new Class[] { TestTool.class, HashFunction.class});

//Usual Job Setup
//...
return 0;
}
```