Package: yarn-dev-tools
YARN-dev-toolsThis project contains various developer helper scripts in order to simplify every day tasks related to Apache Hadoop YARN development.Main dependenciesgitpython- GitPython is a python library used to interact with git repositories, high-level like git-porcelain, or low-level like git-plumbing.tabulate- python-tabulate: Pretty-print tabular data in Python, a library and a command-line utility.bs4- Beautiful Soup is a Python library for pulling data out of HTML and XML files.TODO: Missing dependenciesContributingTODOAuthorsSzilard Nemeth-Initial work-Szilard NemethLicenseTODOAcknowledgmentsTODOGetting startedIn order to use this tool, you need to have at least Python 3.8 installed.Use yarn-dev-tools from package (Recommended)If you don't want to tinker with the source code, you can downloadyarn-dev-toolsfrom PyPi as well.
This is probably the easiest way to use it.
You don't need to install anything manually as I created ascriptthat performs the installation automatically.
The script has asetup-varsfunction at the beginning that defines some environment variables:These are the following:YARNDEVTOOLS_ROOT: Specifies the directory where the Python virtualenv will be created and yarn-dev-tools will be installed to this virtualenv.HADOOP_DEV_DIRShould be set to the upstream Hadoop repository root, e.g.: "~/development/apache/hadoop/"CLOUDERA_HADOOP_ROOTShould be set to the downstream Hadoop repository root, e.g.: "~/development/cloudera/hadoop/"The latter two environment variables is better to be added to your bashrc / zshrc file (depending on what shell you are using) to keep them between the shells.Use yarn-dev-tools from sourceIf you want to use yarn-dev-tools from source, first you need to install its dependencies.
The project root contains a pyproject.toml file that has all the dependencies listed.
The project uses Poetry to resolve the dependencies so you need toinstall poetryas well.
Simply go to the root of this project and executepoetry install --without localdev.
Alternatively, you can runmakefrom the root of the project.Setting up handy aliases to use yarn-dev-toolsIf you completed the installation (either by source or by package), you may want to define some shell aliases to use the tool more easily.
In my system, I havethese.
Please make sure to source this script so that the command 'yarndevtools' will be available since it's defined as a function.
It is important to specifyHADOOP_DEV_DIRandCLOUDERA_HADOOP_ROOTas mentioned above, before sourcing the script.After these steps, you will have a basic set of aliases that is enough to get you started.Setting up yarn-dev-tools with Cloudera CDSWInitial setupUpload the initial setup scripts to the CDSW files, to the root directory (/home/cdsw)initial-cdsw-setup.shinstall-requirements.shCreate a new CDSW session.
Wait for the session to be launched and open up a terminal by Clicking "Terminal access" on the top menu bar.Execute this command:~/initial-cdsw-setup.sh user clouderaThe script performs the following actions:Downloads the scripts that are cloning the upstream and downstream Hadoop repositories + installing yarndevtools itself as a python module.
The download location is:/home/cdsw/scriptsPlease note that the files will be downloaded from the GitHub master branch of this repository!clone_downstream_repos.shclone_upstream_repos.shExecutes the script described in step 2.
This can take some time, especially cloning Hadoop.
Note: The individual CDSW jobs should make sure for themselves to clone the repositories.Copies thepython-based job configsfor all jobs to/home/cdsw/jobsAll you have to do in CDSW is to set up the projects and their starter scripts like this:ProjectStarter script locationArguments for scriptJira umbrella data fetcher (Formerly: Jira umbrella checker reporting)scripts/start_job.pyjira-umbrella-data-fetcherUnit test result aggregatorscripts/start_job.pyunit-test-result-aggregatorUnit test result fetcher (Formerly: Unit test result reporting)scripts/start_job.pyunit-test-result-fetcherBranch comparator (Formerly: Downstream branchdiff reporting)scripts/start_job.pybranch-comparatorReview sheet backport updaterscripts/start_job.pyreview-sheet-backport-updaterReviewsyncscripts/start_job.pyreviewsyncUse-casesExamples for YARN backporterTo backport YARN-6221 to 2 branches, run these commands:yarn-backport YARN-6221 COMPX-6664 cdpd-master
yarn-backport YARN-6221 COMPX-6664 CDH-7.1-maint --no-fetchThe first argument is the upstream Jira IDThe second argument is the downstream Jira ID.The third argument is the downstream branch.The--no-fetchoption is a means to skip git fetch on both repos.How to backport to an already existing relation chain?Go to Gerrit UI and download the patch.
For example:git fetch "https://gerrit.sjc.cloudera.com/cdh/hadoop" refs/changes/29/156429/5 && git checkout FETCH_HEADCheckout a new branchgit checkout -b my-relation-chainRun backporter with:yarn-backport YARN-10314 COMPX-7855 CDH-7.1.7.1000 --no-fetch --downstream_base_ref my-relation-chainwhere:The first argument is the upstream Jira IDThe second argument is the downstream Jira ID.The third argument is the downstream branch.The--no-fetchoption is a means to skip git fetch on both repos.The--downstream_base_ref <local-branchis a way to use a local branch to base the backport on so the Git remote name won't be prepended.Finally, I set up two aliases for pushing the changes to the downstream repo:alias git-push-to-cdpdmaster="git push <REMOTE> HEAD:refs/for/cdpd-master%<REVIEWER_LIST>"
alias git-push-to-cdh71maint="git push <REMOTE> HEAD:refs/for/CDH-7.1-maint%<REVIEWER_LIST>"where REVIEWER_LIST is in this format: "r=user1,r=user2,r=user3,..."ContributingSetup of pre-commitConfigure precommit as described inthis blogpost.Commands:Install precommit:pip install pre-commitMake sure to add pre-commit to your path. For example, on a Mac system, pre-commit is installed here:$HOME/Library/Python/3.8/bin/pre-commit.Executepre-commit installto install git hooks in your.git/directory.Running the testsTODOTroubleshootingInstallation issuesIn case you're facing a similar issue:An error has occurred: InvalidManifestError: 
=====> /<userhome>/.cache/pre-commit/repoBP08UH/.pre-commit-hooks.yaml does not exist
Check the log at /<userhome>/.cache/pre-commit/pre-commit.log, please run:pre-commit autoupdateMore info can be foundhere.