Package: data-eng-utils
Documentation for the package (copy paste it into Chrome/Edge)file://fa1rpwapxx272/ons/ci/jenkins_home/userContent/de-utils/build/index.htmlIntroductionde_utilsis a package that contains collections of useful functions grouped by theme e.g. HDFS, Spark, Python, etc... and a data engineering framework under theengineering_utilssub-package.This is an open-source project designed to be used by anyone and does not follow the needs of a specific project or team. The functions in here will be generalised for wider use, we will not make deprecations or backward incompatible changes unless absolutely necessary, which would be marked with a (major release)[https://www.geeksforgeeks.org/introduction-semantic-versioning/].How to get involvedTo suggest an improvement/ change or contribute code please raise an issue or submit a merge request.You don't have to submit the finished product, it can be:A function that is not fully built because you're either stuck on a problem or just to collaborate with others on buildingAn idea for a function you need, it might be that someone else already has built that function or is willing to help build itGitlab issue example- As shown it's worth keeping discussions around a GitLab issue in the comments section of the issue, instead of a private chat on teams, to allow others to get involved.Developer contribution guideSetup SSHThis only needs to be done once across all CDSW projects so skip this if you've already done this.Open CDSW and navigate to the settings tab using the menu on the left sideClick on theoutbound SSHtab and copy your user public SSH keyOpen GitLab and navigate to preferences using the dropdown menu in the top right of the screenClick on theSSH Keystab in the menu on the left side of the screenPaste your SSH key and give it a relevant title, e.g.cdswClickAdd keySetup environmentClone the project by creating a new CDSW project and (using git for the initial setup with the repo's SSH link)Open the workbench and start a sessionSetup your environment variables for pip and ArtifactoryOpen the terminal and run:pip3 install -r requirements-dev.txtpre-commit installto install the pre-commit hooks (flake8,pydocstyle&darglint) used in development to ensure coding style consistencyCode Style GuidePlease follow the style of the existing codebase, which followsNumPy documentation stylefor Python code and docstrings.The code should follow standardPEP8 guidelinesFurther reading:how to write beautiful Python code.Contributions should also:Use native Spark functions over UDFs where possibleUse()for multi-line code instead of\line continuation charactersUse absolute imports and no wildcard imports i.e.import pandas as *Import pyspark functions as FBe designed to use the user's existing Spark sessionDocument any library dependencies in the requirements.txtDevelopment workflowCreate feature branch off of the development (dev) branch and name it after the function or group of work being submittedgit checkout dev,git pull origin dev&git checkout -b your_branchCommit work regularlygit add -p filenameorgit add filename&git commit -m 'commit message'&git push originhint:git add -p ...allows you to cycle through individual changes before committingThe pre-commit hooks will fail the commit if Flake8 or pydocstyle find issues with the code.Each error will have a description and a code which can be googled for further info.If you have created any new functions or classes make sure that they are publically exposed.In the module levelinit.py file, import them and add a string representation of their name toall.If you have created a new public module, include this in theinit.py in the parent directory.Ensure all private scripts and functions are named with a leading underscore.Further reading around the use of the __init__.py file.Once the ticket is finished create a merge request for 'your_branch' into developmentGo to GitLab to manually create a merge request, selecting the option that says "delete source branch once merge request is accepted"Merge request needs to be peer-reviewed by at least one colleague and have one 👍  before the merge request is acceptedThis means the changes have been carefully reviewed and the code is ready for mergingAmend code in line with any comments from the peer review if neededOnce merged into development the core developers will perform a final review of submissions and merge development into masterWhen merging to master the package version number will be bumped up in line withsemantic versioningPeer-reviews - things to look out for and ask yourself when reviewing code:Check code for any bugs and that the logic makes senseCan I easily understand what the code does?If I had to fix this, would I understand enough?Is the code sufficiently documented for me to understand it and does it explain the why? -"Code Tells You How, Comments Tell You Why"Do the naming conventions make sense/ follow PEP8 and are they consistent with the rest of the project?Have any dependencies changes been added and the requirement.txt updated?Further reading on this topicGitLab email notifications:These are very useful but to save you from being distracted by GitLab emails from peer reviews and comments, create a rule to divert emails fromgitlab@ons.gov.ukinto their own folder. The easiest way to do this is:Select an email from GitLab and find the rules dropdown on the main options bar in Outlook.Click on always move emails from GitLab.A window will open with a tree view of all your Outlook folders, select the main one right at the top and click the "New" button.Be nice 😀We'd like our contributors to feel valued, so if you have found something useful in this repo either a function or a new way of doing something please let them know! they have used their time to share hard earnt knowledge and we'd like them to come back.(git blame shows the contributor - top right corner of each file on GitLab)