Release management

Note: this guidelines are work in progress, and release mangers have the right to change this to whatever they prefer. For now this will serve as a documentation place for the tasks involved in release management. 
== The Release Manager role ==
The Release Managers (RM) performs the task of maintaining the main branch of Koha, which becomes the next major release of Koha every 6 months. The RM does this by applying patches for bug fixes and enhancements from Bugzilla that have been set to "Passed QA" by a member of the QA team. The newly elected RM makes the final decision about what bugs will make it into the release. They also set the schedule and announce the deadlines for the release. 
The RM is elected some weeks before a major release to serve for the next release cycle. 
== A new beginning ==
The RM will begin their tenure by setting the database minor version to the next number. For example, if the past RM just released 24.11.00.000, the new RM will commit a new db version of 24.12.00.000 to denote the beginning of the next development version of Koha. 
Guidelines on pushing this commit to main can be found below in Pushing to main.
See how to write new Database_updates.
You can find example in those files:
 ls installer/data/mysql/db_revs/|grep 00000.pl
The choice of cheeky comment is left to the new RM.
You will also want to set the last stable branch to 'protected' on gitlab
== Daily maintenance workflow ==
Release Managers start by checking the list of bugs in the community Bugzilla that have "Passed QA". Typically an RM will start by pushing bug fixes before pushing new features. It is traditional for a new RM to push only bug fixes during the first month of the RM's tenure, so that these fixes can easily be backported to the current stable release of Koha. 
=== Applying the patch ===
Once the RM has selected a good candidate bug for inclusion in main, the RM will apply that bug's patches to a local copy of main. The 'git qa' alias found in Tips and tricks will help an RM do this quickly and easily. 
If there is a conflict, You will have 2 options: 
*Solve the conflict if it's straight forward.
*Set the status of the bug to "Patch doesn't apply" and ask the author of the patch for a rebase.
=== Checking for regressions ===
Next, you should run some tests. For example, if the patch modifies several templates, you should at least run: 
 perl xt/tt_valid.t
 perl xt/author/valid-templates.t
Do some testing, and keep in mind you should be confident in the QA team. 
=== Updating the DB Revision ===
==== Move the database update and set the version ====
If the patch set makes changes to Koha's database data or database schema, the RM will need to make a DBrev commit. 
* Increment the Koha version number in Koha.pm. Example: $VERSION = "24.12.00.000"; changed to: $VERSION = "24.12.00.001"; 
* The database is usually a file in the atomicupdates. Git move the file to the db_revs directory using the version number from Koha.pm: git mv installer/data/mysql/atomicupdate/bug_xxxx.pl installer/data/mysql/db_revs/241200001.pl 
** 24 is the first part of the major version number. 
** 12 is the second part of version number, while .05 and .11 denote stable versions, .06 and .12 denote development versions. 
** 00 is the minor release version. This won't ever change for main. 
** 001 represents the database version. 
* Test the database update with updatedatabase
* Commit the new db_rev file and the change to Koha.pm. 
==== Update the schema files ====
Warning: The RM should ensure the db schema of their test system matches the community schema exactly. Failure to do so will introduce improper Schema files into the community code base.
If the database update changes the database schema, the RM must update Koha's Schema files using the script misc/devel/update_dbix_class_files.pl. 
In koha-testing-docker:
$ dbic
This deletes/creates an empty database called dbic in which it loads the kohastructure.sql file. It then uses it to regenerate the schema files.
Manual way:
$ mysql -uroot -e 'DROP DATABASE IF EXISTS dbic; CREATE DATABASE dbic'
$ cd /home/vagrant/kohaclone
$ mysql -uroot dbic < installer/data/mysql/kohastructure.sql
$ misc/devel/update_dbix_class_files.pl --db_name dbic --db_user root
==== Commit the changes ====
Commit the changes you have made, and give the commit a description such as:
Bug XXXX: DBRev 24.12.00.001
=== Sign the patches ===
Once everything is solved, and testing proves everything is fine with the applied patches, you should sign all the new commits.
This git alias will allow you to just do git s <number of commits>
 s = "!f() { c=`expr $1 - 1`; git filter-repo --message-callback 'return message + b\"\\nSigned-off-by: Full Name <email>\"' --refs HEAD~$c^..; }; f"
You may need to install git filter-repo first.
=== Pushing to main ===
Once everything looks ready to go, it's time to push the new commits to main 
 git push gitmain@git.koha-community.org:koha.git HEAD:main
==== Create a remote for pushing ====
It's recommended to set 2 remotes, one for pushing to main and a separate one for pushing to main-security: 
git remote add upstream git@git.koha-community.org:Koha-community/Koha.git
git remote add security git@git.koha-community.org:Koha-community/security.git
And then push using this shorter version:
git push upstream
git push security <local-security-branch>:main-security
=== Update Bugzilla ===
This is very important: let people know you pushed the patches to main. 
Use the release-tools to automate these steps: 
* Add a comment on the relevant bugs like this "This patch has been pushed to main, thanks $author"
* Change the status to Pushed to main
* Add the version to the Version(s) released in list
# Add the text you want to etc/user.yaml 
release-tools --shell
koha-push updatebz --range HASH..HEAD
koha-push updatebz --range HEAD~12..HEAD # last 12 commits
=== Dependency changes ===
==== Perl ====
It is not uncommon that a patch set adds new dependencies. 
This is the workflow: 
* Patch writer 
** Explains why the new dependency is required and provides information how it was picked on the bug report.
** Updates the entries in the ./cpanfile file. 
** Adds dependency keyword to Bugzilla. 
** Adds Packaging manager/team in CC in Bugzilla 
* Packaging manager/team 
** Is required to sign-off on the bug before it moves further.
** Will take care of packaging if required and provide the packages via the koha-staging repository.
* Release manager
** Tests patches as usual.
** Makes sure all required dependencies are installed.
** Updates the debian/control file and verifies changes before commiting:
ktd --shell
sudo apt-get update
sudo apt-get install apt-file
apt-file update
debian/update-control
** When this happens, the ??? needs to be updated to reflect this situation. The debian/control file is created out of the debian/control.in file, by the debian/update-control script. This script relies on the output of the apt-file
** 
* Automatism..
**Once the patches have been pushed to a branch, a rebuild of KTD will be automatically triggered. [FIXME]
**Once the patches have been pushed to a branch, a job to send bugzilla a debian/control patch will be automatically triggered. [FIXME]
==== Yarn ====
If package.json changes as result of changes to dependencies: RM needs to generate and commit the yarn.lock file with yarn install.
If you are in koha-shell of koha-testing-docker:
sudo rm -rf /kohadevbox/koha/node_modules /kohadevbox/node_modules/
sudo yarn install --modules-folder /kohadevbox/node_modules
prove xt/verify-yarnlock.t
* Commit yarn.lock
* ktd images need to be rebuilt. See: Pipeline on gitlab Reach out to @joubu or @tcohen when in doubt about this step.
== Releasing ==
=== Dependencies ===
Before each feature release, it would be nice to update any outdated libraries we include.
* - Mottie Keyboard releases - Download the release, unpack it, replace the content of koha-tmpl/intranet-tmpl/lib/keyboard/ with the content of the 'dist' of the release and finally squash the files within /languages and /layouts (cat *.js > all.min.js)
=== Schedule ===
See Release maintenance - Releasing as the steps for Release Maintainers and Release Managers are very similar where it concerns the release process!
== Additional tools ==
During 17.11 and 18.05 release cycle, Jonathan Druart added few tricks to simplify and enhance RM workflow.
=== git pre-push hook ===
There is a pre-push hook documented in the 'Tips & Tricks' page.
=== Mark bug report as pushed to main ===
You will need to install and setup pybugz:
https://wiki.koha-community.org/wiki/Pybugz_configuration
I have modified bugz to support --pushed, see the commit on my github
The following script will update the status of the bug reports that have just been pushed to main.
My workflow is:
git remote -v
# RM ssh://gitmainr@git.koha-community.org/koha.git (fetch)
# RM ssh://gitmain@git.koha-community.org/koha.git (push)
# origin git://git.koha-community.org/koha.git (fetch)
# origin git://git.koha-community.org/koha.git (push)
git remote update # origin and RM are in sync
git checkout -B main RM/main
git bz apply XXXXX # Apply patches
# Review and test patches
git so X # Sign off patches
git push RM main # Will be rejected if the git hook catches something wrong
perl ../rm-tools/mark_as_pushed.pl
Here is the content of mark_as_pushed.pl:
use Modern::Perl;
use IO::Prompt;
my $comment = q|Pushed to main for 18.05, thanks to everybody involved!|; # FIXME Adapt this
my @bug_numbers = `git log origin/main..HEAD --oneline | perl -lape 's/^\\S*\\sBug\\s(\\d+).*/\$1/' | sort | uniq`;
chomp for @bug_numbers;
my @existing_authors = `git log origin/main--format="%aN" --reverse|sort|uniq`;
chomp for @existing_authors;
my @current_authors = `git log origin/main..HEAD --format="%aN" --reverse|sort|uniq`;
chomp for @current_authors;
my @new_authors = array_minus( \@current_authors, \@existing_authors );
if ( @new_authors ) {
 say "\n\n========= NEW AUTHORS ===========";
 say "\t $_" for @new_authors;
 say "=================================\n\n";
}
say "Going to mark the following bug as 'Pushed to main'";
for my $bn ( @bug_numbers ) {
 unless ( $bn =~ m|^\d+$| ) {
 say "WARNING - Bug '$bn' is not a valid bug number, skipping";
 next;
 }
 say "- $bn";
}
my $mark_as_pushed = prompt('OK?: ');
exit 0 unless $mark_as_pushed eq 'yes';
print "\n";
for my $bn ( @bug_numbers ) {
 unless ( $bn =~ m|^\d+$| ) {
 say "WARNING - Bug '$bn' is not a valid bug number, skipping";
 next;
 }
 my $bugz_output = `bugz --connection Koha --skip-auth get -n $bn`;
 my ($status) = ($bugz_output =~ /\nStatus *: (.*)/);
 if ( $status ne 'Passed QA' ) {
 say "========= ERROR ================";
 say " Bug $bn is not Passed QA";
 say "================================";
 next;
 }
 say "Marking bug $bn as pushed to main";
 # --pushed is a modified version of bugz
 # Cannot use bugz from debian repo
 # Use 0.14 from git, with python3
 my $c = `bugz --connection Koha modify $bn --pushed -c "$comment"`;
 say $c; # Search for 'Error:'
}
sub array_minus { # From Array::Utils
 my %e = map{ $_ => undef } @{$_[1]};
 return grep( ! exists( $e{$_} ), @{$_[0]} );
}
=== Generate the pre-release notes ===
I have defined a zsh function release_notes
release_notes () {
 if test $# -ne 2
 then
 echo "Usage: release_notes commit_from commit_to (`release_notes v17.11.00 HEAD`)"
 return
 fi
 from=$1
 to=$2
 export koha_release_notes_dir=../koha-tools/koha-release-notes
 ../koha-tools/release-tools/bin/koha-release $from..$to notes
 koha_version=`perl -MKoha -e "print Koha::version();"`
 commit_id=`git rev-parse --short $to`
 commit_message="$koha_version $commit_id"
 mv "misc/release_notes/`ls -tr misc/release_notes/ | tail -n 1`" $koha_release_notes_dir
 git -C $koha_release_notes_dir commit -a -m"$commit_message"
}
You will certainly want to edit some paths.
You may need to install the following perl modules:sudo apt install libtemplate-perl libwww-mechanize-perl libyaml-syck-perl libjson-perl libyaml-perl librest-client-perl libtext-multimarkdown-perl libhtml-tableextract-perlWhen something is pushed to main, use it asrelease_notes v17.11.00 HEAD
It will generate the release notes from 17.11.00 to HEAD (HEAD should be your current main branch).
A new commit will be added to the koha-release-notes repository
For the first commit each release cycle you will need to add the file (it will be untracked) and commit before pushing. After that the commit should be automatic (you still need to push)
To make it works, you will need to clone the release-tools repo, edit etc/config.yaml with the new release team, and fill the kohadocs entry with a local path to the koha-manual repo
Note: While running the rolling release notes the version for the team needs to be set to dev version - i.e. 18.06 for 18.11
=== Release day actions/checks ===
https://tree.taiga.io/project/koha-ils/task/106
Category:Development
Category:Release_Management