Project: DolphinScheduler (Orchestration & Management / Scheduling & Orchestration)
File: configuration.md

You could use `Resource Center` to upload text files, UDFs and other task-related files. You could configure `Resource Center` to use distributed file system like (2.6+), cluster or remote storage products like , , etc. You could configure `Resource Center` to use local file system. If you deploy `DolphinScheduler` in `Standalone` mode, you could configure it to use local file system for `Resource Center` without the need of an external `HDFS` system or `S3`. Furthermore, if you deploy `DolphinScheduler` in `Cluster` mode, you could use to mount `S3` or to mount `OSS` to your machines and use the local file system for `Resource Center`. In this way, you could operate remote files as if on your local machines. DolphinScheduler Resource Center uses local file system by default, and does not require any additional configuration. But please make sure to change the following configuration at the same time when you need to modify the default value. If you deploy DolphinScheduler in `Cluster` or `Pseudo-Cluster` mode, you need to configure `api-server/conf/common.properties` and `worker-server/conf/common.properties`. If you deploy DolphinScheduler in `Standalone` mode, you only need to configure `standalone-server/conf/common.properties` as follows: The configuration you may need to change: Change `resource.storage.upload.base.path` to your local directory path. Please make sure the `tenant resource.hdfs.root.user` has read and write permissions for `resource.storage.upload.base.path`, e,g. `/tmp/dolphinscheduler`. `DolphinScheduler` will create the directory you configure if it does not exist. NOTE: LOCAL mode does not support reading and writing in distributed mode, which mean you can only use your resource in one machine, unless use shared file mount point Please modify the value of `resource.storage.upload.base.path` if you do not want to use the default value as the base path. The local config is `resource.storage.type=LOCAL` it has actually configured two setting, `resource.storage.type=HDFS` and `resource.hdfs.fs.defaultFS=file:///`, The configuration of `resource.storage.type=LOCAL` is for user-friendly, and enables the local resource center to be enabled by default if you want to upload resources to `Resource Center` connected to `S3`, you need to configure `api-server/conf/common.properties` and `worker-server/conf/common.properties`. You can refer to the following: config the following fields ```properties ...... resource.storage.type=S3 ...... resource.aws.access.key.id=awsaccesskey_id resource.aws.secret.access.key=awssecretaccess_key resource.aws.region=us-west-2 resource.aws.s3.bucket.name=dolphinscheduler resource.aws.s3.endpoint= ...... ``` After version 3.0.0-alpha, if you want to upload resources to `Resource Center` connected to `HDFS`, you need to configure `api-server/conf/common.properties` and `worker-server/conf/common.properties`. ```properties data.basedir.path=/tmp/dolphinscheduler resource.storage.type=LOCAL resource.storage.upload.base.path=/tmp/dolphinscheduler resource.aws.access.key.id=minioadmin resource.aws.secret.access.key=minioadmin resource.aws.region=cn-north-1 resource.aws.s3.bucket.name=dolphinscheduler resource.aws.s3.endpoint=http://localhost:9000 resource.alibaba.cloud.access.key.id=<your-access-key-id> resource.alibaba.cloud.access.key.secret=<your-access-key-secret> resource.alibaba.cloud.region=cn-hangzhou resource.alibaba.cloud.oss.bucket.name=dolphinscheduler resource.alibaba.cloud.oss.endpoint=https://oss-cn-hangzhou.aliyuncs.com resource.huawei.cloud.access.key.id=<your-access-key-id> resource.huawei.cloud.access.key.secret=<your-access-key-secret> resource.huawei.cloud.obs.bucket.name=dolphinscheduler resource.huawei.cloud.obs.endpoint=obs.cn-southwest-2.huaweicloud.com resource.hdfs.root.user=hdfs resource.hdfs.fs.defaultFS=hdfs://mycluster:8020 hadoop.security.authentication.startup.state=false java.security.krb5.conf.path=/opt/krb5.conf login.user.keytab.username=hdfs-mycluster@ESZ.COM login.user.keytab.path=/opt/hdfs.headless.keytab kerberos.expire.time=2 resource.manager.httpaddress.port=8088 yarn.resourcemanager.ha.rm.ids=192.168.xx.xx,192.168.xx.xx yarn.application.status.address=http://ds1:%s/ws/v1/cluster/apps/%s yarn.job.history.status.address=http://ds1:19888/ws/v1/history/mapreduce/jobs/%s datasource.encryption.enable=false datasource.encryption.salt=!@#$%^&* data-quality.jar.dir= support.hive.oneSession=false sudo.enable=true development.state=false alert.rpc.port=50052 conda.path=/opt/anaconda3/etc/profile.d/conda.sh task.resource.limit.state=false appId.collect: log ``` Note: If only the `api-server/conf/common.properties` file is configured, then resource uploading is enabled, but you can not use resources in task. If you want to use or execute the files in the workflow you need to configure `worker-server/conf/common.properties` too. If you want to use the resource upload function, the deployment user in must have relevant operation authority. If you using a Hadoop cluster with HA, you need to enable HDFS resource upload, and you need to copy the `core-site.xml` and `hdfs-site.xml` under the Hadoop cluster to `worker-server/conf` and `api-server/conf`, otherwise skip this copy step.