Project: DolphinScheduler (Orchestration & Management / Scheduling & Orchestration)
File: development-environment-setup.md

Before setting up the DolphinScheduler development environment, please make sure you have installed the software as below: : v1.8.x (Currently does not support jdk 11) : v3.5+ : v16.13+ (dolphinScheduler version is lower than 3.0, please install node v12.20+) : v6.x Download the git repository through your git management tool, here we use git-core as an example ```shell mkdir dolphinscheduler cd dolphinscheduler git clone git@github.com:apache/dolphinscheduler.git ``` Supporting system: MacOS Liunx Run `mvn clean install -Prelease -Dmaven.test.skip=true` DolphinScheduler uses `Spotless` for code style and formatting checks. You could run the following command and `Spotless` will automatically fix the code style and formatting errors for you: ```shell ./mvnw spotless:apply ``` We also have provided a `pre-commit` config file for easy configuration. To use it, you need to have python installed and then install `pre-commit` by running the following command: ```shell python -m pip install pre-commit ``` After that, you can run the following command to install the `pre-commit` hook: ```shell pre-commit install ``` Now, every time you commit your code, `pre-commit` will automatically run `Spotless` to check the code style and formatting. DolphinScheduler will release new Docker images after it released, you could find them in . If you want to modify DolphinScheduler source code, and build Docker images locally, you can run when finished the modification ```shell cd dolphinscheduler ./mvnw -B clean package \ -Dmaven.test.skip \ -Dmaven.javadoc.skip \ -Dspotless.skip = true \ -Ddocker.tag=<TAG> \ -Pdocker,release ``` When the command is finished you could find them by command `docker images`. If you want to modify DolphinScheduler source code, build and push Docker images to your registry <HUB_URL>you can run when finished the modification ```shell cd dolphinscheduler ./mvnw -B clean deploy \ -Dmaven.test.skip \ -Dmaven.javadoc.skip \ -Dspotless.skip = true \ -Dmaven.deploy.skip \ -Ddocker.tag=<TAG> \ -Ddocker.hub=<HUB_URL> \ -Pdocker,release ``` If you want to modify DolphinScheduler source code, and also want to add customize dependencies of Docker image, you can modify the definition of Dockerfile after modifying the source code. You can run the following command to find all Dockerfile files. ```shell cd dolphinscheduler find . -iname 'Dockerfile' ``` Then run the Docker build command above You could create custom Docker images base on those images if you want to change image like add some dependencies or upgrade package. ```Dockerfile FROM dolphinscheduler-standalone-server RUN apt update ; \ apt install -y <YOUR-CUSTOM-DEPENDENCE> ; \ ``` _Note_ Docker will build and push linux/amd64,linux/arm64 multi-architecture images by default Have to use version after Docker 19.03, because after 19.03 docker contains buildx There are two ways to configure the DolphinScheduler development environment, standalone mode and normal mode : Recommendedmore convenient to build development environment, it can cover most scenes. : Separate server master, worker, api, which can cover more test environments than standalone, and it is more like production environment in real life. _Note:_ Use standalone server only for development and debugging, because it uses H2 Database as default database and Zookeeper Testing Server which may not be stable in production. Standalone is only supported in DolphinScheduler 1.3.9 and later

Standalone server is able to connect to external databases like mysql and postgresql, see for instructions. Use different Git branch to develop different codes If you want to develop based on a binary package, switch git branch to specific release branch, for example, if you want to develop base on 1.3.9, you should choose branch `1.3.9-release`. If you want to develop the latest code, choose branch branch `dev`. Find the class `org.apache.dolphinscheduler.StandaloneServer` in Intellij IDEA and clikc run main function to startup. Install frontend dependencies and run it. Note: You can see more detail about the frontend setting in . ```shell cd dolphinscheduler-ui pnpm install pnpm run dev ``` The browser access address can login DolphinScheduler UI. The default username and password are admin/dolphinscheduler123 Download , and extract it. Create directory `zkData` and `zkLog` Go to the zookeeper installation directory, copy configure file `zoo_sample.cfg` to `conf/zoo.cfg`, and change value of dataDir in conf/zoo.cfg to dataDir=./tmp/zookeeper ```shell dataDir=/data/zookeeper/data dataLogDir=/data/zookeeper/datalog ``` Run `./bin/zkServer.sh` in terminal by command `./bin/zkServer.sh start`. The DolphinScheduler's metadata is stored in relational database. Currently supported MySQL and Postgresql. We use MySQL as an example. Start the database and create a new database named dolphinscheduler as DolphinScheduler metabase After creating the new database, run the sql file under `dolphinscheduler/dolphinscheduler-dao/src/main/resources/sql/dolphinscheduler_mysql.sql` directly in MySQL to complete the database initialization Following steps will guide how to start the DolphinScheduler backend service Open project: Use IDE open the project, here we use Intellij IDEA as an example, after opening it will take a while for Intellij IDEA to complete the dependent download File change If you use MySQL as your metadata database, you need to modify `dolphinscheduler/pom.xml` and change the `scope` of the `mysql-connector-java` dependency to `compile`. This step is not necessary to use PostgreSQL Modify database configuration, modify the database configuration in the `dolphinscheduler-master/src/main/resources/application.yaml` Modify database configuration, modify the database configuration in the `dolphinscheduler-worker/src/main/resources/application.yaml` Modify database configuration, modify the database configuration in the `dolphinscheduler-api/src/main/resources/application.yaml` We here use MySQL with database, username, password named dolphinscheduler as an example ```application.yaml spring: datasource: driver-class-name: com.mysql.cj.jdbc.Driver url: jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 username: dolphinscheduler password: dolphinscheduler ``` Log level: add a line `<appender-ref ref="STDOUT"/>` to the following configuration to enable the log to be displayed on the command line `dolphinscheduler-master/src/main/resources/logback-spring.xml` `dolphinscheduler-worker/src/main/resources/logback-spring.xml` `dolphinscheduler-api/src/main/resources/logback-spring.xml` here we add the result after modify as below: ```diff <root level="INFO"> <appender-ref ref="STDOUT"/> <appender-ref ref="APILOGFILE"/> </root> ``` _Note:_ Only DolphinScheduler 2.0 and later versions need to inatall plugin before start server. It not need before version 2.0. There are three services that need to be started, including MasterServer, WorkerServer, ApiApplicationServer. MasterServerExecute function `main` in the class `org.apache.dolphinscheduler.server.master.MasterServer` by Intellij IDEA, with the configuration VM Options `-Dlogging.config=classpath:logback-spring.xml -Ddruid.mysql.usePingMethod=false -Dspring.profiles.active=mysql` WorkerServerExecute function `main` in the class `org.apache.dolphinscheduler.server.worker.WorkerServer` by Intellij IDEA, with the configuration VM Options `-Dlogging.config=classpath:logback-spring.xml -Ddruid.mysql.usePingMethod=false -Dspring.profiles.active=mysql` ApiApplicationServerExecute function `main` in the class `org.apache.dolphinscheduler.api.ApiApplicationServer` by Intellij IDEA, with the configuration VM Options `-Dlogging.config=classpath:logback-spring.xml -Dspring.profiles.active=api,mysql`. After it started, you could find Open API documentation in http://localhost:12345/dolphinscheduler/swagger-ui/index.html The `mysql` in the VM Options `-Dspring.profiles.active=mysql` means specified configuration file Install frontend dependencies and run it ```shell cd dolphinscheduler-ui pnpm install pnpm run dev ``` The browser access address can login DolphinScheduler UI. The default username and password are admin/dolphinscheduler123