Project: Doris (App Definition and Development / Database)
File: README.md

<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> This tool is used to get the progress of all current table transitions during the online `segment_2` function. Currently, you can specify 3 dimensions (in the conf file) to view the results, you can specify one of them individually, or you can customize the combination (that is, specify multiple at the same time). We use MySQLdb python lib to fetch meta from FE, so you must install it. You can get MySQLdb lib from https://pypi.python.org/pypi/MySQL-python, then you can install it as follows: ``` $ tar zxvf MySQL-python-*.tar.gz $ cd MySQL-python-* $ python setup.py build $ python setup.py install ``` Fill in the conf according to your cluster configuration, and specify the table or be you want to watch. Execute `python showsegmentstatus.py` If you want to watch the process of a table named `xxxx`, you can specify `table_name = xxxx` in conf file If you want to watch the process on be whose be_id is `xxxx`, you can specify `be_id = xxxx` in conf file The first number is about segment v2, the latter one is the total count. ``` ==========SUMMARY()=========== rowset_count: 289845 / 289845 rowsetdisksize: 84627551189 / 84627551189 rowsetrowcount: 1150899153 / 1150899153 =========================================================== ==========SUMMARY(table=xxxx)=========== rowset_count: 289845 / 289845 rowsetdisksize: 84627551189 / 84627551189 rowsetrowcount: 1150899153 / 1150899153 =========================================================== ==========SUMMARY(be=10003 )=========== rowset_count: 79650 / 79650 rowsetdisksize: 24473921575 / 24473921575 rowsetrowcount: 331449328 / 331449328 =========================================================== ``` r Set `--service` to `ali`, and then test connectivity on Huawei Cloud. ``` sh emr_tools.sh --case ping --endpoint oss-cn-beijing-internal.aliyuncs.com --region cn-beijing --service ali --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log ``` Alibaba Cloud EMR also supports testing connectivity for both Doris with DLF metadata and Doris on OSS-HDFS storage. The DLF metadata connectivity test needs to be performed on the EMR cluster where the DLF serves as the metadata store, Default value of `DLFENDPOINT` is `datalake-vpc.cn-beijing.aliyuncs.com`, configured at

To test the OSS-HDFS storage connectivity, need to , Default value of `JINDOENDPOINT` is `cn-beijing.oss-dls.aliyuncs.com`, configured at pingtest/ping_poc.sh. ``` sh emrtools.sh --profile defaultemr_env.sh ``` Or Set `--service` to `tx`, and then test connectivity on Huawei Cloud. ``` sh emr_tools.sh --case ping --endpoint cos.ap-beijing.myqcloud.com --region ap-beijing --service tx --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log ``` ``` sh emrtools.sh --profile defaultemr_env.sh ``` Or Set `--service`to `hw`, and then test connectivity on Huawei Cloud. ``` sh emr_tools.sh --case ping --endpoint obs.cn-north-4.myhuaweicloud.com --region cn-north-4 --service hw --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log ``` When the `--case` option is set to `data_set`, will test the query performance of Doris external table: `--test` test data set: ssb, ssb_flat, tpch, clickbench and all. Default `all`. `--service`, EMR cloud vendors: ali(Alibaba), hw(Huawei), tx(tencent). `--host`, Doris Mysql Client IP. `--user`, Doris Mysql Client Username. `--port`, Doris Mysql Client Port. Just modify the above environment variable in `defaultemrenv.sh`, the script will execute `source defaultemrenv.sh` to make the environment variable take effect. If environment variables are configured, you can run the test script directly with the following command: ``` sh emrtools.sh --profile defaultemr_env.sh ``` To run the standard test set using the `emr_tools.sh` script, you need to rewrite the object storage bucket specified by the `BUCKET` variable, and then prepare data in advance and put them under the bucket. The script will generate table creation statements based on the bucket. Now the `emrtools.sh` script supports iceberg, parquet and orc data for ssb, ssbflat, tpch, clickbench. After the connectivity test, the Doris Catalog corresponding to the standard test set is created Prepare the test set data based on the object storage bucket specified by the `BUCKET` variable Generate Spark table creation statements and create Spark object storage tables on EMR Create the spark table in the local HDFS directory: `hdfs:///benchmark-hdfs` You can choose to analyze Doris tables ahead of time and manually execute the statements in `analyze.sql` in the Doris Catalog Execute standard test set scripts: `runstandardset.sh` Full test. After executing the test command, Doris will run ssb, ssb_flat, tpch, clickbench tests in sequence, and the test results will include the cases on HDFS and on the object storage specified by `--service`. ``` sh emrtools.sh --case dataset --service ali --host 127.0.0.1 --user root --port 9030 > log ``` Specify a single test. `--test` option can be set to one of ssb, ssb_flat, tpch and clickbench. ``` sh emrtools.sh --case dataset --test ssb --service ali --host 127.0.0.1 --user root --port 9030 > log ```