Kim Greene

Michelle Smith

kim@kimgreene.com

msmith@kimgreene.com
Paul Albrightpaul.albright@hcl-software.com

1

Kim Greene - Introduction
 Owner of an IT consulting company
 Kim Greene Consulting, Inc.
 www.kimgreene.com

 Started my career at IBM, left and launched my own

business in 2000
 Focus areas:
 HCL collaboration software portfolio Customers are worldwide and in multiple industries
 Blog: www.dominodiva.com
 Twitter: iSeriesDomino
2Michelle Smith - Introduction
 Owner of an IT consulting company
 M.E.Smith Consulting Started fixing IT issues during University work terms

and learned I loved solving the ‘puzzles’ First exposure to Lotus Notes and Domino was

migrating a non-Y2K compliant email environment toDomino… wow that was a while ago!
 Focus areas:
 HCL collaboration software portfolio

 Work with Kim to support a group of amazing clients!
3

Paul Albright - Introduction
 Technical team lead for Application Development support team at HCL
 Supporting HCL Notes, HCL Domino, HCL Enterprise Integrator, HCL

Domino Rest API, HCL Domino LEAP, HCL LEAP
 I hit 25 years supporting these products this month Bachelor of Science degree in Computer Science Master of Science in Management specializing in Computer

Information systems

4Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

5

Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

6

And so it Begins
 You start your day … and then “oh sh!+” happens
 People panic
 OMG
 Phones are ringing
 Details coming in are vague
 Confusion about what the issue is Incorrect conclusions are made
 Sound familiar?
7

Sometimes it is the smallest details
 Case is opened as “Intermittent issue sending emails”
 Mail routing SWE and customer troubleshoot mail routing
 Finally on a call with customer Customer mentions “only happens when clicking button inapplication”
 Look at code behind button
 Found and resolved issue Knowing that “little detail” would have saved days of

work

8

Need to Look at the Big Picture Customer problem
 Every time I flush my toilet my computer restarts
 Checked all of the standard things
 Chatting with customer
 Found out lives in a trailer Computer and electric toilet on same circuit
 Extra load from the “flush” caused the issue

9That Problem is Happening Again Get pinged by customer
 “Hey, remember that lookup issue, it’s happening again”
 Started troubleshooting Application “A”
 Screen share, customer said “oh sorry, it’s not that DB, it’s

this one”

10Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

11

Scoping the Problem
 Key items to gather
 Who is impacted?
Where are they located?
 How many people are impacted?
 Internal users, external users, both?
 Any executives involved?
 How is the business affected?

Loss of revenue
 Employees unable to do their jobs?
12Scoping the Problem
 Key items to gather (cont’d) What platforms are impacted?


While it may be “obvious to you”, HCL Support won’t know theplatform Which protocols are impacted?
 NRPC HTTP SMTP POP3
 LDAP

13

Scoping the Problem
 Key items to gather (cont’d)
 Has anything changed?
Operating system
 Domino version
 Notes version
 Applications
 Network Firewall

14

Scoping the Problem
 Key items to gather
 Screen captures and videos are your friends!

 Screen share session with affected user (avoid the

middleman) Is it reproducible?
 What are the steps needed to reproduce?
15

Scoping the Problem
 ALL of this is important to understand “even if you

don’t think they have anything to do with the issue”

16Scoping the Problem
 Example – User A of a Notes workflow application is notable to send auto-generated emails to User B

 User A has been using the application since 2022
Assumption: User would have reported the issue sooner

 When did the problem start?

Restated: When was the last time this worked?
 The answer may just be that it never worked17

Scoping the Problem
 Example (cont’d) Screen capture of error sentAh, so an error is viewed in the Notes client The resolution was found when searching through Forumcomments – more on that later in our Troubleshooting
section!

18

Scoping the Problem
 Example - Global business has offices in China, India, andthe US During certain times of the day, users in China and India can

not connect to the server via Notes or Web Browser
Domino servers are hosted in US
 Log review shows server is up during this time

 Hosting provider suggested: Setting tasks to run at different times during the night China / India users run scripts to test connectivity
 Assumption: “It’s the great firewall of China”
 Therefore, a network issue

19

Scoping the Problem
 Example (cont’d) Key Item missed in scope - Does it happen in the US thattime of the day?
This was missed because no one reported an issue at thatlocation. Why would that be?
This was about 3 AM Central time No one used their email at that time

We finally set our alarm clocks and tested at that time, anddetermined the issue also affected users in the US

Scope changed significantly!

 Issue eventually found to be nightly jobs being performed

in Data Center of the hosting company

20

Scoping the Problem
 Example – Issue happens randomly during the day, for aperiod of about 30 minutes

 Users receive delivery failures for emails sent externally Users receive incoming emails with .EML attachments

rather than being properly formatted

21

Scoping the Problem
 Example (cont’d) When does the issue occur?
Initially, the time of the issue was determined by the time of calls to the
Support Desk To determine exact times, reviewed the log.nsf around the reportedtime Found message “File Cannot Be Created” during these times
 Searched log.nsf for the error What else is happening when issue occurs?
 Found the client was also having issues with backups around thesetimes.
 Correlating the exact time of the errors in the log with the time of
the backups helped narrow down the issue.
22

Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

23

Enable Logging BEFORE Problems Happen
 Core debug that needs to always be in place Semaphore debug to have enabled

DEBUG_THREADID=1


Helpful to identify process or thread holding a semaphore

DEBUG_CAPTURE_TIMEOUT=1
 DEBUG_SHOW_TIMEOUT=1
 DEBUG_SEM_TIMEOUT=XUsed to specify how long a semaphore must timeout before being
reported24

Enable Logging BEFORE Problems Happen
 Deletion logging
 Very helpful in determining “who/what” did the deleting Example – documents missing in shared database

Invoices sent to a mail-in database External process “POPs” messages from the database Multiple Notes client users of the database Messages have gone missing Enabled Deletion Logging
Quickly identified user who was HARD deleting documents from Trash

25

Data to Collect on a Regular Basis Statistics
 Gather statistics for system as a whole Capture statistics on:

CPU Memory Disk I/O Network I/O Some examples










Perfmon
Perfpmrnmon
vmstatCollection Services
iostatnetstat

26

Data to Collect on a Regular Basis Statistics – Example - Collection Services

27

Data to Collect on a Regular Basis Activity logging
 Records user activity by:
Person Database Access protocol Great for debugging performance slowdowns and CPU

spikesSpecific application or protocol caused CPU spike or performanceslowdown
 Spike in user activity is cause of CPU spike or performanceslowdown28

Data to Collect on a Regular Basis Message Tracking
 Tracking of email messages passing through a Domino
server Use to determine what happened to a message
Was it delivered, read, deleted?
 When did it arrive?
 Etc.
 Easy to Configure: Configuration Settings -> Router/SMTP -> Message Tracking
 Restart the ROUTER task

29

Enable Logging BEFORE Problems Happen Where are application errors / log events written to?
 Console
Check console log sizes to reduce change of wrapping Console_log_max_kbytes=300000
 Mirroring logs
 Console_Log_Mirror=1 Separate logging database Makes problem analysis much more efficient More details provided in the Troubleshooting Essentials section
 Agent logs
 It’s not just about logging, but what “not” to log
 More details provided in the Troubleshooting Essentials section30

Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

31

Troubleshooting Essentials
 Breathe
 Don’t panic!
 Don’t hide embarrassing data Can significantly delay problem resolution
 Remember, they’ve seen it allLike going to the doctor

32

Troubleshooting Essentials
 Be methodical in approach Don’t Forget the basics – some issues ARE simple


Example - MarvelClient
Customer utilizes panagenda MarvelClient to push Connection
documents to users
 New Notes client installations not receiving Connection
documents
 Review of MarvelClient configuration shows should be working
 … however the license had expired33Troubleshooting Essentials
 Listen to your intuition
 Example – application issues
When reviewing applications, often drawn to particular part of design Usually that is where the issue is occurring Example – customer said it was the SAN, my intuition said it was

the network

Customer had intermittent performance issue on their mail server
 Mail server used a SAN
 Mail server was “supposed” to have a 10 GB connection to the SAN
 VM where Domino server was controlled had multiple NIC cards
 Most NIC cards were 1 GB, only 2 were 10 GB
 App running on VM controlled security cameras
 Host was configured to use a round robin policy

34Troubleshooting Essentials
 Finger pointing doesn’t help
 Favorite ‘finger pointing’ conclusion:‘It Must be the Network’Warning: The above only works if you are not also the ‘Network
Person’ Rule out everything you can, it’s not hot potato
 Focus should be solving the problem

35Troubleshooting Essentials
 Check for common solutions
 Google
Include “HCL Domino” or “HCL Notes”
 Use exact error message
 Search your internal KB
 What, you don’t have one? You need one!
 Example - Notes Client issue
 User reports getting error when opening Notes that client“configuration hasn’t been fully setup”
 HelpDesk support personnel just installed the Notes client They didn’t follow their documentation on setting up/configuring
the Notes client for the user

36

Troubleshooting Essentials
 Check for existing solutions
 Search HCL KB articles
 Example - “File does not exist”

Tip: “open” a Case with the error
message shows documents thataren’t “googleable”

37Troubleshooting Essentials
 Run maintenance on impacted databases
 Why?
Corruption problems can quickly be remedied HCL Support will ask you to run these

38

Troubleshooting Essentials
 Run maintenance on impacted databases
 How?
load fixup -F path\filename.nsf
load compact –c –i –d path\filename.nsfNOTE: Add -J if Transactional Logging is enabled on the server
Copy style compact won’t complete if anything accesses the DB

load updall –R path\filename.nsf

 Tip: Include the results of the maintenance in yourdocumentation

39

Reviewing Logs
 Key to know specifics on dates and times problem

happened/happens Particularly important when reviewing logs from different

systems (a.k.a. non-Domino)

Example – “File cannot be created” error seen in log.nsf whenusers are unable to send external emails

Gathered times of the occurrences Used 3rd party tool consoleEZ to search log for the error Once we had the times, we were able to correlate the errors to
the time of the start of the backups

40

Reviewing Logs
 Check appropriate logs
 Domino Notes XPages
 HTTP Agent
…

41

Reviewing Logs
 XPages These files are in the data\domino\workspace\logs
directory on the Domino server or Notes clienterror-logs and trace-logs

 Content of files help locate what XPage is causing the issue,

but the errors often help point to the portion of the XPageto look closer at for the issueFor example, any Java errors will include class having the issue
and that allows you to just look at the parts using that particular
class

42

Reviewing Logs
 XPages Error-log
 <CommonBaseEvent creationTime="2024-01-09T16:01:03.678-05:00"
globalInstanceId="EL0a866a8700018cefcdcb0800000006" msg="CLFAD0134E:
Exception processing XPage request" severity="50" version="1.0.1"><extendedDataElements name="CommonBaseEventLogRecord:level"
type="noValue"><children
name="CommonBaseEventLogRecord:name" type="string"><values>SEVERE</values></children></extendedDataElements><extendedDataElements
name="CommonBaseEventLogRecord:sourceClassName" type="string"><values>com.ibm.commons.log.AbstractLogMgr</values></extendedDataElements><extendedDataElements
name="CommonBaseEventLogRecord:sourceMethodName" type="string"><values>log</values></extendedDataElements><extendedDataElements name="CommonBaseEventLogRecord:Exception"
type="string"><values>com.ibm.xsp.exception.EvaluationExceptionEx: Error while executing
JavaScript action expression&#xD;&#xA;&#x9;atcom.ibm.xsp.binding.javascript.JavaScriptMethodBinding.invoke(JavaScriptMethodBi
nding.java:126)&#xD;&#xA;&#x9;at43

Reviewing Logs
 HTTP logs
 Example:


Customer reports emails being sent from the VP of HRs email, BUT …
Emails were not sent from the VP

Domlog.nsf shows first login request 11/30/2016 at 1:26 AM

VP doesn’t use webmail

First login shows successful login, password was known How did it happen?


VP of HR used mobile device at coffee shop the day prior SSL was not implemented (yet) on Traveler server

44

Reviewing Logs
 Agent Logs
 Print statements are a common debug method, BUT …
Often left behind and fills up the console.log
 Caution: Often contains sensitive data!!


Tip: Create a Boolean debug value and use it to control if the printstatement should be printed.
 If not actively debugging set it to False
Sub Initialize
Dim debug As Boolean
debug = True ' set to False to disable debug' some LS code
If debug Then
Print "At Place 1 "End IfEnd Sub
45

Reviewing Logs
 Agent Logs (cont’d) Tip 1: Use the NoteLog LotusScript method to debug
Write debug to an email, a text file or a Notes database Keeps the console clean, allows you to look at just the data you
need




https://help.hcltechsw.com/dom_designer/12.0.2/basic/H_NOTESLOG_CLASS.html

 Tip 2: Use OpenLog method to log and debug

OpenNTF project Write errors and log messages to an OpenLog database Works with LotusScript and Java agentshttps://www.openntf.org/main.nsf/project.xsp?r=project/openlog

46

Reviewing Logs
 Example – OpenLog Much easier than combing through console.log

47Troubleshooting Agent Issues
 Enable agent manager debug
 tell amgr debug * Make sure to turn it off once the issue is resolved!
 tell amgr debug off
 Pay attention to what the specific agent message(s) are Are the messages coming from:AMgr Agent Manager
 Agent Message
48

Troubleshooting Agent Issues
 Understanding “AMgr:” messages
 Come from an external process Control of agent has been turned over to another process

Lotus Connector example
AMgr: Agent ('TSCompanyUpdate' in 'corpwrk.nsf') message box:
Rate Lotus Connector Error 12809 on line 69 Error: Invalid field

SQL Connector example
AMgr: Agent ('Assigned Reports' in 'ExternalBus.nsf') message box:
ExternalBus.nsf => CommonFunctions_SL => AssignedReport
Error 91 on line 137 Object variable not set

49

Troubleshooting Agent Issues
 Understanding “Agent Manager:” messages
 Come from the agent manager itself


Examples:

Agent Manager: Error validating execution rights for agent 'Access OS' indatabase 'Pauls.nsf'. Agent signer 'Test User7/HCL', effective user 'Test
User7/HCL'. Examine 'Programmability Restrictions' field in the Server
Record.

Agent Manager: 'RunSourceAgent' in database 'helpdesk/people.nsf'.
Agent signer 'Ray Ozzie/IRIS', effective user 'Ray Ozzie/IRIS'. Agent signer,'Ray Ozzie/IRIS', does not have access to this server.



Agent Manager: Agent 'Access OS2' error: Operation is disallowed in thissession50

Troubleshooting Agent Issues
 Understanding “Agent message:” messages
 Come from the agent
 If in LS code where manipulating forms, views, etc., will comefrom the agent itself


Example:

Agent message: CheckPassword => ProfileAudit Error Handler: No
logs open on line 79

 Tip: Use the LS debugger Can the LS code be run manually from the client?
 If so, enable LS debugger from the Tools menu in the client and hit
continue until the error occurs.
 Line throwing the error is the line highlighted in the debugger.

51

Troubleshooting Performance Issues
 Views
 Limit the number of sorted or categorized columns
 Use hidden views sorted by a single column for look ups
 Use Manual refresh setting in view where possible

 Number of documents
 Can we archive or move documents to different database?
52

Key Also to Review These Logs
 Firewall logs
 Operating System logs
 Other environments and applications connect to DB2
 Other servers
 Activity logging
 Message tracking

53Key Also to Review These Logs
 Firewall logs
 Example:
Users experiencing significant sync delays to mobile devices
weeks after moved to Traveler HA
 All Traveler servers in the pool have Green status All Traveler servers can ping each other Firewall updates made, blocking IPC ports 50125 and 50126
54

Key Also to Review These Logs
 Operating system logs
 Example:
Users reporting very slow response times accessing mail server
 Nothing out of ordinary in console logs
 Port trace to server seems fine
 Reviewing OS logs, see backup from Saturday in an Event Wait state Review of backup logs revealed the problemEnd of volume reach on tape drive

Backups had core Domino files locked, dramatically impacting
performance55

Key Also to Review These Logs
 Other environments and applications connect to DB2
 Other servers
 Example - OpenLog - DB2 errors

56Key Also to Review These Logs
 Other environments and applications connect to Example - OpenLog – DB2 error details

57

Key Also to Review These Logs
 Activity Logging
 Example - Want to disable POP3
on port 110 and only allow
Secure POP3 over port 995
Want to proactively help users
change their settings
 But who is using POP3?
 Need to activate Activity Logging
 Tools -> Analyze -> Activity
Select POP3
 Below we see the 3 users who
have used POP3 in the selected
time range58Key Also to Review These Logs

 Message tracking
 Great for finding the delivery
 To Use: In Domino Administrator client
status of emails

Messaging -> Tracking Center -> New
Tracking Request

Note: The data reindexes periodically (15min by default) so don’t be surprised if yourrecent emails are not shown Example – User reporting they

did not receive an email

59Troubleshooting Essentials
 Look for what lead up to the issue Example – Searches are slower
Customer has custom written tool that searches and removes
malicious emails from users’ mail files
 Customer reports searches are slower than they used to be No errors in the application
 Nothing found in console logs
 No performance issues found with hosting system Analysis of search results reveals searches have been slower since
September 23
 Server was upgraded on September 2260Troubleshooting Essentials
 Complete analysis BEFORE making changes!!!
 Absolutely essential
 Change one thing at a time!!
 Otherwise, what fixed it??

61

Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

62

Documenting Document EVERYTHING!!!
 Screen captures
 Dates/time stamps
 Detailed log of what you:
Have found Have done
 Any changes made
When
 By whom
 The reason the change was made
63

Documenting

64Documenting
 Start right away!
 You WILL forget Even if it “may” not seem relevant, include it Remember, everyone is panicking people are probably
stressed Your documentation will become a formal or informal

knowledge base At least parts of it Not all your notes are needed in the documentation for others –edit out the OMGs 
65

Documenting Notes.ini updates Document who, when, why Example of notes.ini updates:

66

Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

67

Communicating Communicating what is going on and what to expect is key!
 Frequency
 Messaging (change for different audiences …)
 Communicate to: Stake holders
 Business users
 Support desk
 Affected User(s) Example - emails to the organization
 This is a known issue, projected time to fix is 1 hour Tip: Send another email once problem is resolved

68Communicating Communication needs to be Open Honest
 Ongoing Pick up the phone or do a screen share!!
 Call the “customer” / “end user” / “affected person”
 Need to know what the ‘end user’ is seeingAnd most importantly, what the end user is ‘doing’69Communicating Avoid assumptions
 Can cause important information to be overlooked Before you “assume” try this crazy method called “asking”

70Communicating Ask questions
 Don’t just accept what have been told If something doesn’t make sense, get clarification
 Use open ended questions, not just ‘yes’ and ‘no’
 Key in on ‘pauses’ … make them talk!

71

Communicating “Everybody lies”
 Intentionally or notWe are imperfect beingsDon’t take what you are told as fact, you will need to verify

 Example
 “It is happening constantly”
 In reality the issue happened whenever they restarted Notes

 How to get people to “talk”
 “I apologize I didn’t make it more clear / tell you about /…” “Help me understand …”
72

Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

73

Bringing in Outside Help You’ve taken it as far as you can, now what?
 Open a Case with HCL Support
 Bring in your Business Partner or consultant
 If it’s a bug in Notes/Domino/XPages, … code
 Need HCL to fix
 Or find a way to workaround the problem

74

Bringing in Outside Help
 Example - Workaround Server crashing with “PANIC: TLA pointer is null! Has this threadcalled OSInitThread()?” running specific Java agent after upgrade to
12.0.2Java agent line causing crash: private static
ResourceBundle.getBundle("config", java.util.Locale.getDefault());
Prior to Domino 12, HCL never set a Java default
 config.properties attached to agent was utilized Starting in version 12, HCL sets a Java default
 config.properties file referenced is now config_en_US.properties

Workaround:

Change agent to use this line of code:

private static ResourceBundle keyConfig = ResourceBundle.getBundle("config");
75

Bringing in Outside Help
 Tip: Provide as much detail as possible Remember that you …
Live and breathe in your environment every day Know the history of your environment
 Know what has changed / stayed the same Etc …
76

Working with HCL Support
 Proper preparation before opening a Case Collect as much detail as possible
Helps with quicker resolution
 Allows HCL to research the problem more thoroughly If know DB causing the issue
Run maintenance on it Detail what have done so farBe very specific77Working with HCL Support
 Tips for working a case with support
 Be open and honest

We don’t care who caused the issue or what you tell your boss,
we are just trying to resolve the problem

 If you do not understand why support is asking you to dosomething, ask questions
 Keep communication open

If you will not be able to perform a test for a while, let us know
and we can follow up accordingly78What HCL Will Ask For
 Expect to be asked to run some form of the following

commands after opening a case:On your Administrator client > Server > Server Console
Issue the following commands: (wait for each one to finish
before going to the next command)







load fixup fixup -F -J mail\sample.nsf (-F if transactional logging isenabled)load compact -c -D -i mail\sample.nsf
load updall -R mail\sample.nsf

 Same commands we recommended previously
 Run them ahead of time, post to case when opening and save some time
 Tip: you should have this in your documentation, if so, can justupload to the caseAgain, save some time79

What HCL Will Ask For
 For issues affecting the server
 Console log files
 Server and Notes client crashes
 NSD files
 Console log files
 Semaphore debug files
 Java Heap dump if one is created as part of the crash (.php file) XPages or Domino LEAP issues
 Error and trace files
 Errors in custom applications
 Simplified reproducible scenario Non-encrypted design only copy of application
 Details on name of view, agent, form, … having issues with80

What HCL Will Ask For
 Where do I find the files?
 Server files:

Domino\Data\IBM_TECHNICAL_SUPPORT directory

 Notes client files:


Notes\Data\IBM_TECHNICAL_SUPPORT directory XPages or Domino LEAP:Domino\Data\Domino\workspace\logs directory What is their naming convention?

 console.log, console_DomMail_2023_10_04@04_53_42.log, …
 console_DESKTOP-CS8B1S4_2023_11_09@22_11_10.log, …
 nsd_07_19_21@12_06_20.nsd, …
 SEMDEBUG.TXT, SEMDEBUG_DomMail_2023_08_01@04_37_22.TXT HeapDump_20241126.105324.45640.0001.phd
 trace-log-0.xml, …
 error-log-0.xml, …
81

Working with HCL Support
 Lessons Learned
 HCL Ambassadors

Remember to use your template when opening Cases

 Summarize information as Case progressesVery helpful to make sure key information isn’t forgotten or“buried”

 Ask for a screen sharing session to ensure the issue isunderstood82

Working with HCL Support
 Lessons Learned (cont’d) Sometimes you get a new L2
We were all new once (some of us haven’t been for a while) Be Patient

But not to the detriment of the resolution

 If a Case is not getting traction, you can:Raise the priority of the case
 Request a Duty Manager be assigned to review
 Request an escalation of the case83

Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

84

Resolving the Issue Once cause is identified, apply appropriate fix
 Might involve repairing, replacing, updating or configuring
new hardware or software
 Test and Confirm
 Make sure the problem is resolved and the solution iseffective Remove any extra debug or logging

85

Resolving the Issue Update Documentation
 Record the problem and solution for future reference Follow up with the affected person(s) Contact them directly - as support people, when do we getto talk to someone and deliver GOOD news?
 Remember: If you have a Support Case open with HCL, update it86Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

87

rd
Accelerating Troubleshooting - 3 Party Tools
 Free or low cost tools MarvelClient Essentials
https://help.hcltechsw.com/domino/12.0.2/admin/plan_marvel_client.html
 https://www.panagenda.com/kbase/display/mc/MarvelClient+Essentials
 EBCDIC to ASCII converter

UltraEdit https://www.ultraedit.com/downloads/ultraedit-download-thank-you/ Lotus Notes Diagnostic

Also converts EBCDIC to ASCII https://lotus-notes-diagnostic.software.informer.com


88

rd
Accelerating Troubleshooting - 3 Party Tools
 Free or low cost tools
 KillNotes
https://download.cnet.com/killnotes/3000-2094_4-75905088.html

 NotesPeek

https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=
KB0036425
 64-bit version for Domino 14 is available on Flexnet NSFPEEK

https://www.openntf.org/main.nsf/project.xsp?r=project/NSFPeek89

rdAccelerating Troubleshooting - 3 Party Tools
 Free or low cost tools All-in-one admin tool
https://support.hcltechsw.com/csm?id=kb_article&sysparm_articl
e=KB0032494

 Wireshark

https://www.wireshark.org/download.html
 Configuring Wireshark to troubleshoot performance issues


https://support.hcltechsw.com/csm?id=kb_article&sysparm_article=KB0097503

90

rdAccelerating Troubleshooting - 3 Party Tools
 Other tools we can recommend Ytria EZ SuiteAdministration and Development tool sets
Free functionality provided in each toolhttps://www.ytria.com/ezsuite/ MarvelClient Essentials is included in your subscription, additional modules allow for automation of
more features and functionality https://www.panagenda.com/kbase/display/mc/About+panagenda+MarvelClient We use these tools daily with our customers who are licensed to use them
 Why?
Saves $$$ in consulting expenses
Ease and accuracy of automation
Ability to dig deep into problems
More accurate

91

Agenda There’s a Problem, Now What?!?
 Scoping the Problem
 Enabling Debug and Logging
 Troubleshooting Essentials
 Documenting
 Communicating Working with HCL Support and/or Business Partner/Consultant
 Resolving the Issue Accelerating troubleshooting with 3rd Party Tools
 Q&A

92

Questions?

93Contact Informationwww.linkedin.com/in/kimgreeneconsulting@iSeriesDomino

kim@kimgreene.com

94