本地模式

0.7版本后Hive开始支持任务执行选择本地模式(local mode)大多数的Hadoop job是需要hadoop提供的完整的可扩展性来处理大数据的。
在hive中运行的sql有很多是比较小的sql,数据量小,计算量小 ,这些比较小的sql如果也采用分布式的方式来执行,那么是得不偿失的.因为sql真正执行的时间可能只有10秒,但是分布式任务的生成的其他过程的执行可能要1分钟.这样的小任务更适合采用lcoal mr的方式来执行.就是在本地来执行,通过把输入数据拉回客户端来执行.

启用方式

set hive.exec.mode.local.auto=true;(默认为false)
当一个job满足如下条件才能真正使用本地模式:
job的输入数据大小必须小于参数:hive.exec.mode.local.auto.inputbytes.max(默认128MB)job的map数必须小于参数:hive.exec.mode.local.auto.tasks.max(默认4)job的reduce数必须为0或者1

例子

我这里有一张表,其实就是我们在前面教程hive streaming中演示用到的,我先在不开启本地模式下执行一下
select weekday,count(1) from ods.u_data_new group by weekday;
下面是执行日志,我们看到大概花费了15.762 秒
Starting Job = job_1608438780277_0029, Tracking URL = http://localhost:8088/proxy/application_1608438780277_0029/ 2020-12-27 12:51:01,739 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] exec.Task (SessionState.java:printInfo(1227)) - Starting Job = job_1608438780277_0029, Tracking URL = http://localhost:8088/proxy/application_1608438780277_0029/ Kill Command = /usr/local/Cellar/hadoop/3.2.1/libexec/bin/mapred job -kill job_1608438780277_0029 2020-12-27 12:51:01,739 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] exec.Task (SessionState.java:printInfo(1227)) - Kill Command = /usr/local/Cellar/hadoop/3.2.1/libexec/bin/mapred job -kill job_1608438780277_0029 Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2020-12-27 12:51:06,967 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] exec.Task (SessionState.java:printInfo(1227)) - Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1 2020-12-27 12:51:06,984 WARN [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] mapreduce.Counters (AbstractCounters.java:getGroup(235)) - Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 2020-12-27 12:51:06,984 Stage-1 map = 0%, reduce = 0% 2020-12-27 12:51:06,984 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] exec.Task (SessionState.java:printInfo(1227)) - 2020-12-27 12:51:06,984 Stage-1 map = 0%, reduce = 0% 2020-12-27 12:51:11,068 Stage-1 map = 100%, reduce = 0% 2020-12-27 12:51:11,068 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] exec.Task (SessionState.java:printInfo(1227)) - 2020-12-27 12:51:11,068 Stage-1 map = 100%, reduce = 0% 2020-12-27 12:51:16,171 Stage-1 map = 100%, reduce = 100% 2020-12-27 12:51:16,172 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] exec.Task (SessionState.java:printInfo(1227)) - 2020-12-27 12:51:16,171 Stage-1 map = 100%, reduce = 100% Ended Job = job_1608438780277_0029 2020-12-27 12:51:17,198 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] exec.Task (SessionState.java:printInfo(1227)) - Ended Job = job_1608438780277_0029 MapReduce Jobs Launched: 2020-12-27 12:51:17,206 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] ql.Driver (SessionState.java:printInfo(1227)) - MapReduce Jobs Launched: Stage-Stage-1: Map: 1 Reduce: 1 HDFS Read: 1192486 HDFS Write: 227 SUCCESS 2020-12-27 12:51:17,206 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] ql.Driver (SessionState.java:printInfo(1227)) - Stage-Stage-1: Map: 1 Reduce: 1 HDFS Read: 1192486 HDFS Write: 227 SUCCESS Total MapReduce CPU Time Spent: 0 msec 2020-12-27 12:51:17,206 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] ql.Driver (SessionState.java:printInfo(1227)) - Total MapReduce CPU Time Spent: 0 msec 2020-12-27 12:51:17,206 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] ql.Driver (Driver.java:execute(2531)) - Completed executing command(queryId=liuwenqiang_20201227125101_07ce9455-4e6d-40f5-9e33-e9691a0a459d); Time taken: 15.655 seconds OK 2020-12-27 12:51:17,206 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] ql.Driver (SessionState.java:printInfo(1227)) - OK 2020-12-27 12:51:17,206 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] ql.Driver (Driver.java:checkConcurrency(285)) - Concurrency mode is disabled, not creating a lock manager 2020-12-27 12:51:17,209 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] mapred.FileInputFormat (FileInputFormat.java:listStatus(259)) - Total input files to process : 1 2020-12-27 12:51:17,210 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] sasl.SaslDataTransferClient (SaslDataTransferClient.java:checkTrustAndSend(239)) - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false 2020-12-27 12:51:17,211 INFO [680439f2-d65f-4c1f-8eaf-c5aee654b618 main] exec.ListSinkOperator (Operator.java:logStats(1038)) - RECORDS_OUT_OPERATOR_LIST_SINK_10:7, RECORDS_OUT_INTERMEDIATE:0, 1 12254 2 13579 3 14430 4 15114 5 14743 6 18229 7 11651 Time taken: 15.762 seconds, Fetched: 7 row(s)
现在我们开启本地执行模式再执行一下
set hive.exec.mode.local.auto=true; select weekday,count(1) from ods.u_data_new group by weekday;
从下面的日志输出中我们看到了使用了LocalJobRunner也就是启动了本地执行模式,最后的耗时只有 1.507秒,相比分布式的执行模式还是快了不少的,因为很多时候我们做测试啊或者本就数据量不大,这种情况下还是很有用的。