Kafka REST Proxy on Mac

#はじめに
[Kafka REST Proxy](https://github.com/confluentinc/kafka-rest)をMacで使用するにあたり、少しはまったのでメモを残しておきます。

#動作確認環境
OS:macOS Mojave Version 10.14.6 (18G84)

#1. Kafka REST Proxyのビルドでエラー

```
> git clone https://github.com/confluentinc/kafka-rest.git
> cd kafka-rest
> mvn install
```
で以下のようなエラーが発生。

```
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for kafka-rest-parent 5.5.0-SNAPSHOT:
[INFO] 
[INFO] kafka-rest-parent .................................. SUCCESS [  2.062 s]
[INFO] kafka-rest-common .................................. FAILURE [  0.063 s]
[INFO] kafka-rest-scala-consumer .......................... SKIPPED
[INFO] kafka-rest ......................................... SKIPPED
[INFO] kafka-rest-console-scripts ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.575 s
[INFO] Finished at: 2019-12-28T17:44:08+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project kafka-rest-common: Could not resolve dependencies for project io.confluent:kafka-rest-common:jar:5.5.0-SNAPSHOT: Failure to find io.confluent:rest-utils:jar:5.5.0-SNAPSHOT in http://packages.confluent.io/maven/ was cached in the local repository, resolution will not be reattempted until the update interval of confluent has elapsed or updates are forced -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :kafka-rest-common```
```

どうやら、[この記事](https://stackoverflow.com/questions/56388665/could-not-find-artifact-io-confluentkafka-rest-parentpom5-4-0-snapshot-and-p)によると、SNAPSHOTSはローカルには無いので[FAQ](https://github.com/confluentinc/kafka-rest/wiki/FAQ)を見るかConfluent Platformを使えば良いとのこと。

#2. Kafka関連の色々をBuild
ということで、以下のProjectをビルドしローカルのMavenリポジトリに必要なライブラリをインストールします。

| Project | Clone | Build with |
|:--|:--|:--|
| Kafka | https://github.com/confluentinc/kafka/ | ./gradlew installAll |
| Confluent Common library | https://github.com/confluentinc/common | mvn clean install |
| Confluent Rest-Utils | https://github.com/confluentinc/rest-utils | mvn clean install |
| Avro Converter | https://github.com/confluentinc/schema-registry | mvn clean install |


まずはkafka本体。
初回は１時間過ぎてもずっと終わらず、一旦停止。よくわからないが再度実行すると今度は成功。

```
> git clone https://github.com/confluentinc/kafka.git
> cd kafka
> ./gradlew installAll
```



そして、次にConfluent Common libraryをビルド。エラーが発生。。。

```
> git clone https://github.com/confluentinc/common.git
> cd common
> mvn clean install
(中略)
>
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for common 5.5.0-SNAPSHOT:
[INFO] 
[INFO] assembly-plugin-boilerplate ........................ SUCCESS [  1.468 s]
[INFO] Build Tools ........................................ SUCCESS [  1.596 s]
[INFO] common ............................................. SUCCESS [  2.422 s]
[INFO] utils .............................................. SUCCESS [  2.426 s]
[INFO] metrics ............................................ SUCCESS [  3.572 s]
[INFO] config ............................................. SUCCESS [  2.342 s]
[INFO] common-logging ..................................... SUCCESS [  3.647 s]
[INFO] confluent-log4j-extensions ......................... SUCCESS [  2.562 s]
[INFO] confluent-log4j2-extensions ........................ SUCCESS [  2.888 s]
[INFO] package ............................................ FAILURE [  0.905 s]
[INFO] Common ............................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  24.527 s
[INFO] Finished at: 2019-12-28T17:19:13+09:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:3.0.0:single (package-assembly) on project common-package: Execution package-assembly of goal org.apache.maven.plugins:maven-assembly-plugin:3.0.0:single failed: group id '1614999791' is too big ( > 2097151 ). Use STAR or POSIX extensions to overcome this limit -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :common-package
```

Mac特有のエラーのようなので[こちら](https://tutorialmore.com/questions-826377.htm)を参考にtarLongFileModeのオプションを追記し、再度ビルドすると成功。

```pom.xml
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-assembly-plugin</artifactId>
                    <version>${maven-assembly.version}</version>

                    <!-- added this configuration tag -->
                    <configuration>
                     <descriptors>
                      <descriptor>src/assembly/distribution.xml</descriptor>
                      </descriptors>
                      <tarLongFileMode>posix</tarLongFileMode>
                      <outputDirectory>${project.build.directory}</outputDirectory>
                      <appendAssemblyId>false</appendAssemblyId>
                     </configuration>
                </plugin>

```

次はConfluent Rest-Utils。こちらは普通にビルド成功。

```
> git clone https://github.com/confluentinc/rest-utils.git
> cd rest-utils
> mvn clean install 
```

そしてAvro Converter。時間は掛かったがこちらも普通にビルド成功。

```
> git clone https://github.com/confluentinc/schema-registry.git
> cd schema-registry
> mvn clean install 
```

#3. Kafka REST Proxyを改めてBuild
やっと成功。

```
>mvn install
(中略)

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for kafka-rest-parent 5.5.0-SNAPSHOT:
[INFO] 
[INFO] kafka-rest-parent .................................. SUCCESS [  6.480 s]
[INFO] kafka-rest-common .................................. SUCCESS [  5.371 s]
[INFO] kafka-rest-scala-consumer .......................... SUCCESS [ 18.041 s]
[INFO] kafka-rest ......................................... SUCCESS [07:35 min]
[INFO] kafka-rest-console-scripts ......................... SUCCESS [  3.344 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  08:09 min
[INFO] Finished at: 2019-12-28T18:41:40+09:00
[INFO] ------------------------------------------------------------------------
```

#4. CLASSPATHを追加で設定
Kafka REST Proxyを起動するも、

```
Error: Could not find or load main class io.confluent.kafkarest.KafkaRestMain
```

ダメ押しでbin/kafka-rest-run-classを修正

```
for dir in $base_dir/kafka-rest/target/kafka-rest-*-development; do
  CLASSPATH=$CLASSPATH:$dir/share/java/kafka-rest/*
done
```

以下に変更。

```
for dir in $base_dir/kafka-rest/target/kafka-rest-*-SNAPSHOT; do
  CLASSPATH=$CLASSPATH:$dir/share/java/kafka-rest/*
done
```

動くようになったと思ったらlog4jのエラー。

```
log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /kafka-rest.log (Permission denied)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
	at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
	at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
	at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)
	at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
	at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
	at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
	at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
	at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
	at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
	at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
	at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
	at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
	at org.slf4j.impl.Log4jLoggerFactory.<init>(Log4jLoggerFactory.java:66)
	at org.slf4j.impl.StaticLoggerBinder.<init>(StaticLoggerBinder.java:72)
	at org.slf4j.impl.StaticLoggerBinder.<clinit>(StaticLoggerBinder.java:45)
	at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
	at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
	at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:412)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:357)
	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)
	at io.confluent.kafkarest.KafkaRestMain.<clinit>(KafkaRestMain.java:28)
```

とりあえず面倒なので、

```
  if [ -e "$base_dir/config/log4j.properties" ]; then # Dev environment
    KAFKAREST_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/config/log4j.properties"
```

を以下に変更。

```
  if [ -e "$base_dir/config/log4j.properties" ]; then # Dev environment
    KAFKAREST_LOG4J_OPTS="-Dlog4j.configuration=file:$base_dir/config/log4j.properties -Dkafka-rest.log.dir=$base_dir"
```

ようやく使えるようになりました。

#参考URL
https://github.com/confluentinc/kafka-rest/wiki/FAQ
https://stackoverflow.com/questions/56388665/could-not-find-artifact-io-confluentkafka-rest-parentpom5-4-0-snapshot-and-p
https://tutorialmore.com/questions-826377.htm