Kim Greene

Michelle Smith

kim@kimgreene.com

msmith@kimgreene.com

1

Kim Greene - Introduction
 Owner of an IT consulting company
 Kim Greene Consulting, Inc.

 www.kimgreene.com

 Started my career at IBM, left and launched my own

business in 2000
 Focus areas:
 HCL collaboration software portfolio Customers are worldwide and in multiple industries
 Blog: www.dominodiva.com
 Twitter: iSeriesDomino
2Michelle Smith - Introduction
 Owner of an IT consulting company
 M.E.Smith Consulting Started fixing IT issues during University work terms

and learned I loved solving the ‘puzzles’ First exposure to Lotus Notes and Domino was

migrating a non-Y2K compliant email environment toDomino… wow that was a while ago!
 Focus areas:
 HCL collaboration software portfolio

 Work with Kim to support a group of amazing clients!
3

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes4Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes5

ACLs
 Are they really controlling access properly?
 When was the last time you checked??
 Ensure Domino databases are locked down appropriately
 Which databases have:
Default = Manager
 OtherDomainServers = Manager
 Anonymous = Manager
 Anonymous missing6Manager Access Anonymous, Default and OtherDomainServers as Manager
 Not a good idea
 Check, you’ll be surprised at how many databases thisapplies to!
 Check HCL supplied templates too Customer example:
 (Data compliments of Ytria aclEZ)

Apps1

Dev2

Mail1

Mail2

Sametime

Traveler

Anonymous Missing

855

202

346

44

58

18

Anonymous Manager

3

4

2

1

3

1

Default Manager

27

27

54

10

14

9

OtherDomainServers
Manager

141

107

31

19

6

5

7

Anonymous Access Should web users be able to view the database??

 Set “Anonymous” to “No Access” for all databases except

public databases

 Without this anyone can see public calendar entries withinyour mail files!!

Anonymous Missing
 If not present, same level of access as ‘-Default-’

 Customer example:
 (Data compliments of Ytria aclEZ)

Apps1

Dev2

Mail1

Mail2

Sametime

Traveler

Anonymous Missing

855

202

346

44

58

18

Anonymous Manager

3

4

2

1

3

1

Default Manager

27

27

54

10

14

9

OtherDomainServers
Manager

141

107

31

19

6

5

9

Enforce Consistent ACL is Key
 Ensures ACL stays consistent across all replicas
 Protects DBs if pulled at file system level!

10Access to Mail Files Who has Manager or Designer access?
 Common carry-over from OOO agent days
 Manager access, users can delete their mail file
 Change to Editor access

11

Review ACLs Regularly Domain Catalog – Access Control

 Ytria aclEZ

Lists -> by Name

12

Examine Agents
 Analyze agent signers
 You’ll be surprised by what you find, guaranteed!!
 Cross-analyze with Domino server security settings Is the signer in the NAB?
 Which servers are agents set to run on?

13

Agent Signers
 Over time it’s easy to lose track of the quantity of agent

signers
 OpenNTF example before cleanup was done (Data compliments of Ytria agentEZ)

14

Agents – Server Set to Run On
 As servers are spun up, replaced, consolidated – it creates

an interesting mix
 Another OpenNTF example
 (Data compliments of Ytria agentEZ)

15

Administration ECL Functions as template for workstation ECLs
 Always check -Default- and -No Signature- settings

 Often many outdated entries exist

16Administration ECL Actions -> Edit Administration ECL-OR Edit through security settings document

17

Administration ECL
 Warning, be careful when working with the Admin ECL

 ALWAYS use an ID that is already in the ECL!!
 Even just viewing the Admin ECL can cause your ID to be
put in the ECL
18

Protected Groups Prevents accidental deletion of designated “critical”

groups Configured in Directory Profile of the Domino Directory
 Tip: You must edit and save once to become operational

19

Protected Groups Prevent deletion of these groups
 Defaults to LocalDomainAdmins, LocalDomainServers, andOtherDomainServers
 Add others critical to your environment

20

Examine/Update HTTP Password Hash Common to have non-secure HTTP password hashes in use Upgrade HTTP password hash to more secure option

 Step 1: Edit Directory Profile
 Actions -> Edit Directory Profile

21

Examine/Update HTTP Password Hash
 Step 2: Upgrade person documents to use ‘more secureinternet password’22

Web Administration Database Has been deprecated, but may still be “lurking”
 Support withdrawn due to cross-site request forgery attack
vulnerability
 Webadmin.nsf allows administrators to administer Domino via abrowser Recommendations: Remove both webadmin.nsf and webadmin.ntf


Tip: Set DominoNoWebAdmin=1 in the notes.ini to ensure the DB
cannot be recreated if the template still exists on a server

 IF still using webadmin.nsf, ensure Anonymous = No Access23

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes24

Memory – Critical for Optimal Performance Examine the hosting server/VM
 High paging and faulting can dramatically impact performance ofDomino

IBM i exampleDrilling down further we see memory pool involved is where all of Domino
servers are running25

Memory Impact on CPU with optimal memory allocation

26

Network Impact on Notes Clients
 Notes client performance can be heavily impacted by

network issues
 Packet loss is of significant importancePacket loss over 4% will have noticeable impact

 Ping with large packet size is your friend ☺


Ex: ping -n 100 -l 8192 <servername or FQIHN>

 Customer example:

Ping statistics for xxx.xxx.xxx.xx:
Packets: Sent = 100, Received = 79, Lost = 21 (21% loss),Approximate round trip times in milli-seconds:
Minimum = 6ms, Maximum = 7ms, Average = 6ms
27

Examine Views
 Views
 Limit the number of sortable columns


Are ALL of the sort options are needed ??

 Use hidden views sorted by a single column for look ups
 Use Manual refresh setting in view where feasible28

Check Inbox Size
 Size of inbox impacts both CPU and end user response time
16

CPU utilization

14
12
10

1,024 docs
4,096 docs
7,196 docs
28,672 docs

8
6
4
2
0
100 MBAvg response times (msec)

5,000 docs in Inbox

1,000 docs in Inbox

45
40
35
30
25
20

700 MB400

1,000

Average mail file size (MB) Tip: Check the Inbox size in the Log.nsf under Usage -> by Sizeand open the document to see details
29

Control Inbox Size
 Configure in server document
 Server Tasks -> Administration Process tab Or via Policy using Mail Settings Document
 Mail -> Basics tab

30

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes31

Have You Tuned Full Text Indexes?
 By default, view updates and full text indexing are driven

by the same thread Prevent long full text indexing operations from delaying

view updates
 Spawn full-text indexing off to own threadUpdate_Fulltext_Thread=1

 Review key FTI statistics:
FT.Index.Count
56,150

FT.Index.Search
27,652

FT.Search.Total.Results
1,002,317
32

Customer FTI Example
 Semaphore timeouts happening
Semaphore # of times Description
Timeout
occurring
3A05
0266
0244
0931
03A8

793
20
15
8
2

SEM_HTTP_AGENT
NSF per-database full-text semaphore
NSF per-database semaphore
Task sync semaphore
DBCONT page zero semaphore Enabling Update_Fulltext_Thread=1 eliminatedsemaphore timeouts

33Are You Preventing FTI Issues?
 Rebuild FTIs periodically with Updall –x option
 Run regularly, can prevent FTI issues
Remedies corruption
 Reduces the size of the FTI Example: On a DB with 77k documents and 1.5 GB, here is

the before and after of the Updall –x:

34

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes35

Checking Traveler Health How often do you check the health of your Traveler servers?
 Let’s be honest ☺
 tell traveler status is your friend
 Green : No issues
 Yellow : Possible issue that should be investigated Red : Critical issues that should be addressed The problem information for Yellow and Red status isincorporated in the command output

36

Traveler Tips
 tell traveler status Example Yellow status
Traveler: The Traveler task has been running since Thu May 02 16:13:19 CDT 2024.
Traveler: The Traveler availability index is currently 98 while servicing 331 users.
Traveler: Database Connection URL: jdbc:as400://10.1.1.1/LNT.
Traveler: Database Connection Properties: user=TRAVELER, clientUser=TRAVELER,
currentLockTimeout=60, loginTimeout=null.
Traveler: The last successful device sync was on Tue Aug 13 13:24:19 CDT 2024.
Traveler: Yellow Status Messages
Traveler: The peak number of HTTP connections is 515 percent of the 437 available HTTP threads.
Traveler: The overall status of Traveler is Yellow.

37

Does Traveler Have Enough Memory?
 tell traveler mem
Traveler: CPU and Memory (MB) Usage History
Traveler: Date
CPU Pct Java Mem C Mem Avl Indx # Users # Errors # DB Conn
Traveler: 2024-08-12 13:02:48 CDT 1.60
917
3139
99
330
30326 1
Traveler: 2024-08-12 13:17:48 CDT 2.17
967
3134
98
330
30326 1…
Traveler: Current Memory Usage
Traveler: Java Memory Usage
Traveler: Max Total 2048 MB
Traveler: Current Total Allocated 1912 MB
Traveler: Available 812 MB (40 percent of Max Total)
Traveler: Used 1236 MB (60 percent of Max Total)
Traveler: C Memory Usage
Traveler: Allocated 3253 MB (20 percent of Total Physical)
Traveler: Current Usage
Traveler: Java 1236 MB
Traveler: C 3253 MB

38Monitoring Traveler Performance Easy to check once and forget Threads
 HTTP threads = # of devices X 1.2


True for Derby and HA implementations

 Traveler thread pools Check GetThreadDelayTime

Want majority in 0-2 second range

 Monitor device synchronization return codes
 503 – Server busy, indicates insufficient synchronization threads

39

Monitor Traveler Performance HTTP threads
 CAREFUL
Allocating too many active HTTP threads will result in
unnecessary server memory consumption
 Ensure system has sufficient resources to handle increased
threads
 HTTP thread consumes
 ~ 1 MB of memory on 32-bit system ~4 MB of memory on 64-bit system40

Defrag Traveler Database Over time Traveler performance can deteriorate
 Defrag the database to restore performance Use DBMAINT to defrag traveler database Tell traveler dbmaint set interval 7 **
 11/19/2017 09:37:02 Traveler: DB maintenance will be performed every 7 days.

 Tell traveler dbmaint set time 23:00
 11/19/2017 09:39:58 Traveler: Time of day for DB maintenance has been set to 23:00
 Tell traveler dbmaint set day Sunday 11/19/2017 09:51:40 Traveler: Day is now configured to Sunday.
 Tell traveler dbmaint set auto on ** 11/19/2017 10:12:27 Traveler: Automatic maintenance of your database has been set.
 11/19/2017 10:12:27 Traveler: The next maintenance is scheduled for 2017-11-22 23:00.
 11/19/2017 10:12:27 Traveler: Maintenance will be performed every 7 days at 23:00.

** Only options available for Derby database41

Can You Wipe the Device?
 Being able to remove data or “wipe” a device is critical Lost or stolen device
 Employee resignation or termination
 Wipe options
 Full or Traveler only
 User or server initiatedUser can only initiate
a wipe if enabled in
the server document,
Notes Traveler tab
42

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes43Deletion Stubs
 Domino’s way of tracking the life cycle of a document
 Eventually deleted when “purge interval” is reached Many deletion stubs can impact performance to end user
 Searches read deletion stubs and include them in response

returnedSearch count only shows count of full documents, which is what usersees

 Use ‘show database dbname.nsf’ to get live and deleted documentcount Important to purge deletion stubs on regular basis
 Purge happens upon DB open in Notes client or DBMT runs

44

Monitor Size of Logging Databases
 Common for agents to write to logging databases as

they process data
 Keep logging databases small
 Semaphore timeout issues and HTTP thread locking
issues with large logging databases Recommend archiving / purging documents out of

logging databases on regular basis

45

Control Log File Sizes
 How large are log.nsf, domlog.nsf and custom log DBs?
 Select to replicate out documents over X days old

46

Corrupt NSFs and NLOs
 Server crashes and other events can cause corruption of NSFs

and NLOs

 Are you proactively managing these?
 DBCapture Tool for non-clustered environments
File named to .cor and moved to IBM_TECHNICAL_SUPPORT folder
 Enable via notes.ini DATABASE_CAPTURE_ENABLED=1
 Repair utility if using cluster symmetry
 Upon database open event, if corruption is detected runs fixup
 IF fixup is unsuccessful: Replace with “good” replica Rename to dbfilename.pd_bad_YYYYMMDDHHMMSSSS
 Tip 1: Ensure AutoRepair and RprCleanup (or RepairCleanup depending
on Domino release) are added to SERVERTASKS= Tip 2: Schedule database cleanup and repair in Program documents
47

Still Using Old Compact Switches?
 Important to check -b and -B were recommended for years-b (recover unused space) -B (recover unused space & reduce file size) Most recent recommend is to use DBMT or compact with

option -C
 Compact -C

Equivalent to a Windows or relational DB “Defragment”
 Rearrange database or reduce file size
 Tip: No longer need debug_enable_compact_8_5=1 to run withmultiple threads

48

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes49

Know Your Schedules
 Have you examined Program document schedules in relation

to backups and other work running on the server?
 i.e. agents, compacts, …

 What is scheduled in Program Documents?
 Are scheduled tasks ending before others kick off?
 Should schedules be adjusted to accommodate backup and agent
schedules?
 Do scheduled tasks conflict with each other?
 What are ServerTasksxx= set to?

50

Know Your Schedules
 ‘show sched’ lists all scheduled tasks

51

Know Your Schedules ‘tell amgr sched’ lists

 Ytria agentEZ

agent schedules, withlimited granularity

52

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes53

Automatic Diagnostic Collection
 Are you collecting crash reports and running fault

analyzer?
 Not enabled by default
 Configuration document → Diagnostics tab

54

Crash Notification Rollup
 Set desktop policy to collect fault reports
 Tip: Set prompt to NO Otherwise security-minded people will cancel

55

Auto Restart After Failure Do your servers have enough time to properly shutdown?
 Some servers need longer than 300 seconds to shutdown Adjust based on your environment
 Are you getting notified when crashes happen?
 Set on Basics tab of the Server document
 Enable ‘Mail Notifications’(shh.. Don’t include your Boss on it)

56

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes

57

Review Server Tasks
 Are critical tasks running?
 Check! Some may be missing
 Not all servers need all tasks
 Calendar Connector

 Message Tracking Collector
 Rooms & Resource Manager
 Schedule manager
 LDAP DECS Collect58

Is RNRMGR Needed?
 Is it needed?
 RNRMGR task (Rooms & Resource Manager)Processes all Rooms and Resources activities Meeting invitations
 Reschedules or cancellations
 Updates Busytime or Clusterbusy database accordingly If used, has the design been updated?
 Make sure you update the design of your Resources
database when upgrading your Domino server
60Old Debug Still Lurking
 Is there lurking debug still enabled?
 Did you really check??
 Consumes valuable resources Make sure your notes.ini doesn’t look like thisDebug_Enable_Update_Fix=8191
Debug_Disable_DDM=1
NSD_RUN_MEMCHECK=1
debug_sem_timeout=1000
DEBUG_PRIVATE_POOLS=1
DEBUG_SHARED_POOLS=1
NSF_DocCache_Thread=1
debug_nif=0
debug_nif_update=1
NSF_DocCache_Thread=1
FT_LIMIT_HIGHLIGHT_FILTER=1
LDAPDEBUG=1
SMTPDebug=361

Notes.ini
 Don’t forget about the configuration document!!

62

Review for Unsupported Variables
 Determine if any unsupported notes.ini variables are in use
 Partial list here:
https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0032795

 Examples:

DEBUG_NAMELOOKUP_MISSES DONT_CACHE_MONITOR_FORMULAS
 LSMODULE_TRACE PercentAvailSysResources (not listed in the KB article, but has been
deprecated )63Do You Have These Variables Set?
 Recommended to be enabled at all times: CONSOLE_LOG_ENABLED=1


Captures server console data and logs to console.log file CONSOLE_LOG_MAX_KBYTES=204800


Restricts the console Log size to 200MB and then overwrites
oldest entries

 DEBUG_CAPTURE_TIMEOUT=1


Captures semaphore time stamp and logs to semdebug.txt

 DEBUG_SHOW_TIMEOUT=1


Captures semaphore information and logs to semdebug.txt

 DEBUG_THREADID=1


Stamps server threads and logs to console.log file64

Prevent Updates to ServerTasksAt Tired of losing your ServerTasksAt customizations whenupgrading?
 SetupLeaveServerTasks to the rescue Add SetupLeaveServerTasks=1 to server’s notes.ini
 Disables automatic updating of the ServerTasks= andServerTasksAtXX= lines during a Domino Server upgrade

65Ensure Proper Event Pool Size
 Especially important if actively debugging a problem
 Message below indicates Event Pool is set too smallWarning: Cannot record event - cannot keep up with event
occurrence rate!

 Increase the event pool size if necessaryDefault size is 10,485,760 bytes. Maximum size is 104,857,600
bytes.

 Add notes.ini EVENT_POOL_SIZE=xxx to notes.ini
 Requires a server restart66

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes67

Is Silent Failover Setup?
 Check your desktop policy settings
 Desktop Settings Document -> Mail tab

68

Is Mail Cluster Failover Configured?

69

Directory Assistance Failover Will directory assistance work during a failover?
 Not if an application link is in use!

 Replace with * for Server Name and specify the filename Tip: Can’t have both for failover to work, the application

link will be used over the servername / filename70

RNRMGR RNRMGR supports clustering, but be aware Only one other server in cluster will be allowed to process
requests
 If the home server (administration server in ACL) is downwhen RNRMGR starts it will not process any requests toprotect cases where the database is out of date from theprimary server
https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB
0088332

 After 30 minutes of down-time, RNRMGR server updates to“first non-home server in line” in the server cluster

Check who owns RNRMGR with this command tell rnrmgr whoowns <resourceDBname>.nsf
71Cluster Symmetry
 Is your configuration optimal?
 Should NOT use on:\IBM_ID_VAULT \MTDATA Does it do what you expect?
 Copy at file system level Only folders below the data directory
 No templates Folder names and filename need to match, including casesensitivity!
72Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes73

Are You Automatically Creating Replicas?
Use a registration policy!

74

Careful with Database Properties
 Quotas and PIRC settings don’t replicate!

 Quotas
 Quotas are unique to a given serverWill only get set on cluster when replica is first created Changes do not replicate You must set the quota on the cluster server to match

quota of file on primary server-OR- Remove quotas on secondary servers

75

Careful with Database Properties PIRC (Purge Interval Replication Control) Prevents old deleted documents from being
recreated/resurrected Enable via database property or with compact commandload compact dbname –pirc on

 Especially important to have enabled on names.nsf
76Missing and Duplicate Replicas
 Missing replicas
 Very common on non-mail servers
 Duplicate replicas
 Very common on mail servers
 How to identify:
 Look at cluster directory database Write your own application to detect Ytria replicationEZ
 Cluster symmetry77Missing and Duplicate Replicas
 Have you looked? You will be surprised!!
 cldbdir.nsf

78

Missing and Duplicate Replicas
 Ytria replicationEZ

79

Is Design Ping-Pong Happening?
Replicates designchanges to 14.0
server

14.0 Server

12.0.2 Server

Updates design of mailfiles to 14.0 design

Updates design of mailfiles to 12.0.2 design
Replicates designchanges to 12.0.2
server80

Preventing Design Ping-Pong Replicating templates between servers
 Most templates share same replica ID across servers
 Latest design gets replicated Stops ping-pong chain of events
 Limiting Design task execution
 Runs by default at 1:00 AM along with Catalog taskServerTasksAt1=Catalog, Design Remove ‘Design’ from this variable set config ServerTasksAt1=Catalog
 Or edit notes.ini file directly Until all servers are at same code level81

Preventing Design Ping-Pong Selective replication
 Choose to not replicate design elements

82

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes83Unread Marks Are unread marks in sync across replicas?
 Have you actually checked??
 Customer example
 Unread mark counters of mail files on primary andsecondary servers(Data compliments of Ytria replicationEZ)
Unread Counts In
Sync

Unread Counts Out of
Sync

No Unread Table on
Secondary Server

955

690

286

49%

36%

15%
84

Unread Marks
 Key to ensure they are properly replicated Review Database properties

85Unread Marks - Creating Replicas
 Key to exchange unread marks during replica creation

 Set ADMINP_EXCHANGE_ALL_UNREAD_MARKS=1 prior tocreating replicas
 Set on the “source” server86Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI & Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes88Pay Attention to Console Errors!!
 Console errors can’t be ignored admin4.nsf has not replicated (PUSH) with ANY server since
MM/DD/YYYY HH:MM:SS (1681 hours ago) admin4.nsf has not replicated (PULL) with ANY server since
MM/DD/YYYY HH:MM:SS (1681 hours ago)
 Error validating execution rights for agent 'Notify' in database‘subdir/dbname.nsf'. Agent signer ‘XXX01/YYY', effective user
‘XXX01/YYY'. Agent signer.
 RnRMgr: The design of Rooms.nsf is not one supportable by RnRMgr.
Autoprocessing is being disabled for this DB.
 Directory Cataloger finished processing DirectoryCatalog.nsf: File does
not exist
 Agent Manager: Full text operations on database ‘mail/myfile.nsf’
which is not full text indexed. This is extremely inefficient.
89

Recertifying Users
 Recertify users BEFORE they expire
 Use the Certification Expiration view in the Domino Directory to see whichusers are due to expire

 Then select users, and click ‘Recertify Selected People’Admin4.nsf – Keep it Clean
 Do not make Test or Development servers part ofyour production Domain Keep it small. Remove old documents
 Up to 21 days is OK
 File -> Properties -> Replication Settings -> Space Savers-> Remove documents not modified in the last # days

 All replicas must have the same setting91Admin4.nsf – Keep it Clean Approve workflow requests
 Renames Delete Mailfiles

 Review documents pending Administrator approval Select ‘Approve Selected Requests’ or edit document to reject92

Agenda Security

 Notes.ini

 Performance Clusters and Failover FTI Health Replication

 Traveler Health Unread Marks Database Health Often Overlooked Schedules‘Necessities’ Q&A Crashes

93Questions?

94

Contact Informationwww.linkedin.com/in/kimgreeneconsulting@iSeriesDomino

kim@kimgreene.com

95