Package: apiarist
ApiaristA python 2.6+ package for defining Hive queries which can be run on AWS EMR.It is, in its current form, only addressing a very narrow use-case.
Reading large text files into a Hive database, running a Hive query, and outputting the results to a text file.File format can be CSV or similar - other delimiters can be specified.The jobs are runnable locally, which is mainly for testing. You will need a local version of Hive which is in yourPATHsuch that the commandhive -f /some/hive/script.hqlcauses hive to execute the contents of the file.It is heavily modeled onmrjoband attempts to present a similar API and use similar common variables to cooperate withboto.A simple Hive jobYou will need to provide four methods:tablethe name of the table that your query will select from.input_columnsthe columns in the source data file.output_columnsthe columns that your query will output.querythe HiveQL query.This code lives in/examples.fromapiarist.jobimportHiveJobclassEmailRecipientsSummary(HiveJob):deftable(self):return'emails_sent'definput_columns(self):return[('day','STRING'),('weekday','INT'),('sent','BIGINT')]defoutput_columns(self):return[('year','INT'),('weekday','INT'),('sent','BIGINT')]defquery(self):return"SELECT YEAR(day), weekday, SUM(sent) FROM emails_sent GROUP BY YEAR(day), weekday;"if__name__=="__main__":EmailRecipientsSummary().run()Try it outLocally (must have a Hive server available):python email_recipients_summary.py -r local /path/to/your/local/file.csvEMR:python email_recipients_summary.py -r emr s3://path/to/your/S3/files/NOTE: for the EMR command, you will need to supply some basic configuration.SerdeHive allows custom a serde to be used to define data formats in tables. Apiarist usescsv-serdeto handle the CSV format properly.This serde also allows configuration of the delimiter, quoting character, and escape character. The defaults are, delimiter =,, quote character =", escape character =\.You can override the defaults in your job. You should be careful about escape sequences when doing so because the value needs to be written into a file.It is best to define them as string literals. Example:fromapiarist.jobimportHiveJobclassEmailRecipientsSummary(HiveJob):INFILE_DELIMITER_CHAR=r'\t'INFILE_QUOTE_CHAR=r"\'"INFILE_ESCAPE_CHAR=r'%'OUTFILE_DELIMITER_CHAR=r'\t'OUTFILE_QUOTE_CHAR=r'\"'OUTFILE_ESCAPE_CHAR=r"\\"ConfigurationThere are a range of options for providing job-specific configuration.Command-line optionsArguments can be passed to jobs on the command line, or programmatically with an array of options. Argument handling uses theoptparsemodule.Various options can be passed to control the running of the job. In particular the AWS/EMR options.-rthe run mode. Eitherlocaloremr(default islocal)--conf-pathuse a YAML configuration file.--output-dirwhere the results of the job will go.--labelAlternate label for the job. Default is job's class name.--ownerWho is running this job (if different from the current user). Default isgetpass.getuser(), orno_userif that fails.--s3-scratch-urithe bucket in which all the temporary files can go.--local-scratch-dirthis is where temporary file will be written.--s3-log-uriwrite the logs to this location on S3.--ec2-instance-typethe base instance type. Default ism3.xlarge--ec2-master-instance-typeif you want the master type to be different.--num-ec2-instancesnumber of instances (including the master). Default is2.--ami-versionthe ami version. Default islatest.--hive-version. Default islatest.--iam-instance-profilerole for the EC2 instances on the cluster. Default isEMR_EC2_DefaultRole.--iam-service-rolerole for the Amazon EMR service on the cluster. Default isEMR_DefaultRole.--s3-sync-wait-timeto configure how long to wait after uploading files to S3.--check-emr-status-everyconfigure the interval between each status check on a running job.--quietless logging--verbosemore logging--retain-hive-tablefor local mode, keep the hive table to run further ad-hoc queries.--visible-to-all-usersmake your cluster visible to all IAM users on the same AWS account. Set by default--no-visible-to-all-usershide your cluster from other IAM users on the same AWS accountNOTE: IAM roles will be mandatory for all users after June 30, 2015. These are set via the--iam-instance-profileand--iam-service-roleoptions above.SeeConfigure IAM Roles for Amazon EMRConfiguration fileYou can supply arguments to your job in a configuration file. It takes the same format asmrjobconfiguration.The name of the arguments is different, using underscores instead of hyphens and omitting leading hyphens.
Config options are divided by the type of runner (local/emr) to allow provision of all options for a job in one file.Below is a sample config file:runners:emr:aws_access_key_id:AABBCCDDEEFF11223344aws_secret_access_key:AABBCCDDEEFF1122334AABBCCDDEEFFec2_instance_type:m3.xlargenum_ec2_instances:5s3_scratch_uri:s3://myjobs/scratchspace/ami_version:3.2.1hive_version:0.13.1local:local_scratch_dir:/home/apiarist/temp/Arguments supplied on command-line or in application code will override those supplied in the config file.Environment variablesSome environment variables are used when the value is not provided in other configuration methods.AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYfor connecting to AWS.S3_SCRATCH_URIa S3 base location where all the temporary file for the job will be written.APIARIST_TMP_DIRwhere local files will be written during job runs. (This is overridden by the--local-scratch-diroption)CSV_SERDE_JAR_S3a permanent location of the serde jar. If this is not set, Apiarist will automatically upload a copy of the jar to an S3 location in the scratch space.Passing options to your jobsJobs can be configured to accept arguments.To do this, add the following method to your job class to configutr the options:defconfigure_options(self):super(EmailRecipientsSummary,self).configure_options()self.add_passthrough_option('--year',dest='year')And then use the option by providing it in the command line arguments, like this:python email_recipients_summary.py -r local /path/to/your/local/file.csv --year 2014Then incorporating it into your HiveQL query like this:defquery(self):q="SELECT YEAR(day), weekday, SUM(sent) "q+="FROM emails_sent "q+="WHERE YEAR(day) ={0}".format(self.options.year)q+="GROUP BY YEAR(day), weekday;"returnqQuerying Hive locallyWhen developing a new query, you may want to fire up Hive to run it and test your syntax.To generate the Hive table, run your job locally with the--retain-hive-tableargument. After it terminates, runhivefrom the command line and you will get a Hive prompt.Becauseapiaristuses a serde to interpret the text files for Hive, you will need to add this serde to the Hive session before your table can be read.The command to do this will be something like:hive> ADD JAR /Users/max/.virtualenvs/apiarist/lib/python2.7/site-packages/apiarist/jars/csv-serde-1.1.2-0.11.0-all.jar;Obviously, your path will be different, depending on where apiarist is installed.Once this is done you can start running interactive HiveQL queries on your text data.LicenseApiarist source code is released under Apache 2 License. Check LICENSE file for more information.