OpenSearch

OpenSearch is the core data storage component of the Observability Logging stack. It replaced ElasticSearch and Kibana in 2021.
==Building==
Easiest path to building locally is to use a container on a Debian Bullseye WMCS instance >= g3.cores2.ram4.disk20
===Environment===
# Install podman and rootlesskit
sudo apt update && sudo apt install -y podman rootlesskit
# Add user subuid/gid settings for ldap users
echo 'subid: sss' | sudo tee -a /etc/nsswitch.conf
# Add registry
echo 'unqualified-search-registries = ["docker.io"]' | sudo tee -a /etc/containers/registries.conf
# Reboot (https://github.com/containers/podman/issues/3890)
sudo reboot
# Once back up and logged in, enable unprivileged_userns_clone
sudo sysctl kernel.unprivileged_userns_clone=1
===OpenSearch===
# Clone repo
git clone https://github.com/opensearch-project/OpenSearch.git
# Enter OpenSearch repo
cd ~/OpenSearch
# Checkout 2.0.0
git checkout tags/2.0.0
# Give podman permissions on source dir
podman unshare chown -R 1000:1000 .
# Enter the container
podman run -it --rm --user gradle --net host --workdir /home/gradle/opensearch -v $(pwd):/home/gradle/opensearch:Z gradle:jdk11 /bin/bash
# Build
./gradlew -Dbuild.snapshot=false distribution:packages:deb:build
# Exit the container
exit
# Move the compiled deb package to home dir
sudo mv distribution/packages/deb/build/distributions/opensearch*.deb ~/
# Optionally change ownership to yourself
sudo chown $(id -u):$(id -g) ~/opensearch*.deb
===OpenSearch Dashboards===
# Clone repo
git clone https://github.com/opensearch-project/OpenSearch-Dashboards.git
cd ~/OpenSearch-Dashboards
# Checkout version 2.0.0
git checkout tags/2.0.0
# Give podman permissions on source dir
podman unshare chown -R 1000:1000 .
# Enter the container
podman run -it --rm --user root --net host -v $(pwd):/home/node/osd:Z node:$(cat .node-version)-buster /bin/bash
# Install dependencies
apt update && apt install -y ruby ruby-dev rubygems build-essential
gem install --no-ri --no-rdoc fpm
# Switch to unprivileged node user
su - node
# Change to build dir
cd /home/node/osd
# Bootstrap
yarn osd bootstrap
# Build
node scripts/build --deb --all-platforms --release
# Exit the container
exit
# Move the compiled deb package to home dir
sudo mv target/opensearch-dashboards*.deb ~/
# Optionally change ownership to yourself
sudo chown $(id -u):$(id -g) ~/opensearch-dashboards*.deb
=== Operations ===
Cookbooks for OpenSearch can be run from cumin hosts.
==== Rolling restarting OpenSearch daemons in a cluster ====
sudo cookbook sre.opensearch.roll-restart-reboot --alias datahubsearch --reason 'operation rationale' restart_daemons
==== Rolling reboot OpenSearch hosts in a cluster ====
sudo cookbook sre.opensearch.roll-restart-reboot --alias datahubsearch --reason 'operation rationale' reboot
===Troubleshooting===
* On a newly provisioned node:
** Failed to create directory or subvolume "/run/opensearch-(...)": Permission denied
*** Run sudo systemd-tmpfiles --create
** No such file or directory - A directory component in /usr/lib/nagios/plugins/check_elasticsearch_unassigned_shards.py(...).lock does not exist or is a dangling symbolic link
*** Run sudo mkdir -p /usr/lib/nagios/plugins