SRE/LDAP

Wikimedia's LDAP configurations apply to logins in Toolforge and Cloud VPSes using sssd. However, some configurations come directly from Puppet's data.yaml.
== LDAP in Production ==
There are two separate LDAP directory trees running in production:
* The main cluster is used for authentication to various developer-oriented services (including Toolforge, Cloud VPS and all services behind CAS-SSO). Accounts on this LDAP tree are called "Wikimedia developer accounts". This cluster has multiple endpoints:
** Writes are sent to a pair of writable servers (one in each main DC, currently seaborgium/serpens). This endpoint is known as 'ldap-rw' (previously 'ldap-labs'), and should only be used by clients that need to change LDAP data.
** Everything else should use the 'ldap-ro' endpoint (backed by multiple ldap-replica* hosts via LVS). These servers have a full replica of the LDAP tree and only exist to increase availability and decrease load from the writable servers.
* cloudservices2XXX-dev hosts run a small cluster that is used like the main cluster on the OpenStack testing environment.
All installations are using OpenLDAP. The writable servers of each cluster are running in mirror_mode, which provides bi-directional replication of write changes using the syncrepl protocol.
== Installing/Configuring the server ==
=== Create indexes ===
After creating the indexes, on each server you need to stop the server, rebuild the indexes, and start the server:
=== Add a proxy agent ===
Since we are requiring authentication by default, the clients need a user to do authenticated lookups. We create a proxyagent user for this:
dn: cn=proxyagent,ou=profile,<basedn>
changetype: add
objectclass: top
objectclass: inetorgperson
objectclass: person
sn: agent
givenName: proxy
userpassword: <aPasswordGoesHere>
cn: proxyagent
== Troubleshooting ==
==== Where to find logs ====
The main daemon logs can be found here:
user@seaborgium:~$ sudo journalctl -u slapd.service
Other interesting logs are:
* TODO: fill me
==== Corrupt nscd cache ====
The nscd cache is a bdb database, and can (rarely) become corrupt when a system shuts down uncleanly. You can forcibly clear the cache by doing the following:
/etc/init.d/nscd stop
rm -f /var/cache/nscd/*
/etc/init.d/nscd start
==== Invalid nscd cache ====
To purge an invalid cache, you can do the following:
nscd -i <database>
Where <database> is passwd, group, or services.
==== Debugging sudo-ldap ====
Sudo will print ldap debugging information, if you add the following to /etc/ldap/ldap.conf:
SUDOERS_DEBUG 2
==== Debugging autofs ====
Autofs will print debugging information to /var/log/syslog, if you add the following to /etc/default/autofs:
LOGGING="debug"
== Common LDAP administrative actions ==
This shows you how to do user administration on LDAP. You need access to mwmaint1002 to perform write actions, and should be either a root or a member of the group ldap-admins. We shall use the wonderful ldapvi program, not invented by us but maintained by a real upstream, to do these tasks.
=== Display extended info about a user ===
[mwmaint1002:~] $ ldapsearch -x uid=$username
Replace $username with the actual user name (uid). You will see the "cn", "sn" and "uid". Note how these can be different. This also displays the email address and SSH key used. Useful to verify if a user has a WMF email address and doesn't have the same key as in production.
=== Show group members / check a user is already a member of a group ===
[mwmaint1002:~] $ ldapsearch -x cn=$grpname
Replace $grpname with the actual group name (e.g. wmf, ops, ...).
=== Find a user by email address ===
[mwmaint1002:~] $ ldapsearch -x mail="foobar@wikimedia.org"
[mwmaint1002:~] $ ldapsearch -x mail="ffoobar*"
This works with an exact email address or wildcards as well. Useful to find a user if the UID they picked is unknown but you know they used their (WMF) email address which can be derived from first and last name.
=== Add a user to a group ===
==== Method 1 ====
[mwmaint1002:~] $ sudo modify-ldap-group wmf ('wmf' is the group name, other valid groups are 'ops', 'wmf', 'nda' or 'wmde')
An editor will open, copy/paste/modify a line, write and quit.
*When adding a user to the wmf LDAP group, please also add them to the Phabricator group wmf-nda (click 'members'->'add members'), see T290605 for background on this).
==== Method 2 ====
ldapvi -b ou=groups cn=$groupname
This brings up an editor with all information about the group, including its list of members. You can add a member by adding the following line to the end of the editor:
member: uid=$shellname,ou=people,dc=wikimedia,dc=org
Then save and exit your editor (<ESC>:wq in vim), you'll be asked if you want to save these LDAP changes. Press y and enter to confirm, or q and enter to cancel
=== Remove a user from a group ===
==== Method 1 ====
[mwmaint1002:~] $ sudo modify-ldap-group wmf ('wmf' is the group name, other valid groups are 'ops', 'wmf', 'nda' or 'wmde')
An editor will open, delete the relevant line, write and quit.
==== Method 2 ====
ldapvi -b ou=groups cn=$groupname
This brings up an editor with all information about the group, including its list of members. You can find the member you want to remove (Press /$shellname to search in vim), remove that line entirely, and then exit and save.
=== Finding the shell name of a user ===
ldapsearch -x -b ou=people,dc=wikimedia,dc=org cn=$wikitechUserName
The shell username will be listed as 'uid', which is also in the DN.
=== Editing info about a user ===
If you want to change details about a user (for a rename, or a ssh key change or password reset), you can do
ldapvi -b ou=people,dc=wikimedia,dc=org uid=$shellusername
Then you can change all the info you want to and save to persist (y to commit changes v if you want to view them first).
Tool accounts have an extra organizational unit servicegroups so to change details about a tool account you can
ldapvi -b ou=people,ou=servicegroups,dc=wikimedia,dc=org cn=tools.$toolname
and proceed the same as per user accounts.
=== Changing the password of a user ===
To change the password of a user (as root, without knowing their current password):
sudo ldappasswd -H ldap://ldap-eqiad.wikimedia.org -x -D "cn=admin,dc=wikimedia,dc=org" -W -S "uid=<USER>,ou=people,dc=wikimedia,dc=org"
You will first be asked twice for a NEW password and then once for the admin password. You can find the admin password in pwstore.
== LDAP structure ==
As of May 2022, the main LDAP directory is structured like this:
* dc=wikimedia,dc=org - directory root
** ou=groups,dc=wikimedia,dc=org - groups, including Cloud VPS projects prefixed with project- and maintained by Keystone
** ou=people,dc=wikimedia,dc=org - User accounts
** ou=sysusers,dc=wikimedia,dc=org - A few special accounts owned by a given team, but not representing a human user, see SRE/LDAP/Sysusers for an overview
** ou=ppolicies,dc=wikimedia,dc=org
*** cn=disabled,ou=ppolicies,dc=wikimedia,dc=org - password policy to disable an account
** ou=profile,dc=wikimedia,dc=org - Some system accounts (including proxyagent for reads and scriptuser for writes)
** ou=projects,dc=wikimedia,dc=org - a mess that contains both stale entries for Cloud VPS project groups and actual data (mostly sudo policies these days) that is actively being read
** ou=servicegroups,dc=wikimedia,dc=org - Toolforge tools and some other special case "service groups"
** ou=sudoers,dc=wikimedia,dc=org - Stale(?) OU for cloud-wide sudoers rules that is not included in the actual config used by Cloud VPS instances
== See Also ==
* LDAP Groups - Rights granted by various LDAP groups
* Renaming users - Renaming a Wikitech/LDAP/Gerrit user
* digitalocean.com community tutorial - raw modifications to an openldap system
* Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta - LDAP account management for WMCS toolsbeta
* Portal:Cloud_VPS/Admin/Testing_deployment#LDAP - LDAP account management for WMCS codfw1dev openstack deployment
* The foxtrot-ldap page, fake LDAP for Toolforge internal development 
Category: Wikimedia infrastructure
Category:Operations