White Paper

Best Practices inOperationalizing ZDXBy Amit Patel
Team Lead, ZDX Deployments© 2023 Zscaler, Inc. All rights reserved.

1. Rollout Strategy

4

1.1 Zscaler client connector version

4

1.2 Pilot rollout

4

1.3 Successive staggered rollout

4

2. Application Strategy

5

2.1 Usage-Based5

2.1.1 Critical applications

5

2.1.2 Group-specific critical applications

5

2.1.3 Noisy applications

6

2.2 Delivery-Based6

2.2.3. VPNs

6

2.2.4 Direct/Zscaler bypassed

6

3. Application arrangement/services

7

4. Probes

7

4.1 Criteria: Run for select users/all

7

4.1.1 Organizations with 30 probes

84.1.2 Organizations with more than 30 probes

84.2 Criteria: DDOS concerns

94.3 Web probe redirect/availability best practices

10

4.3.1 Application that responds with a 200 OK

10

4.3.2 Application that redirects to another service such as login with a 302 Found

10

4.4 Cloud Path best practices

10

4.4.1 Protocol

10

4.4.2 Packet count11© 2023 Zscaler, Inc. All rights reserved.

White Paper

ContentsWhite Paper

2

5. Admin Access/RBAC

11

5.1 Platform Owner

12

5.2 HelpDesk/ServiceDesk

12

5.3 Network Operations12

5.4 T1 Roles

12

6. Alerting13

6.1 Condition values

13

6.2 Nesting13

6.3 Throttles: Violating Count, Devices, and Group By

14

6.3.1 Violating Count

14

6.3.2 Impacted Devices

14

6.3.3 Active Devices

14

6.3.4 Group By

14

6.4 Tweaking/soak period

15

6.5 ITSM/IM integration with webhook

15

6.6 ITSM Routing

16

7. Data analysis

17

7.1 Looking at metrics cohesively than subjectively

17

7.2 Looking at surrounding data rounds, spike/pattern

17

7.3 UCAAS Monitoring18© 2023 Zscaler, Inc. All rights reserved.

White PaperContentsWhite Paper3

1. Rollout strategy
The first step in rolling out Zscaler Digital Experience (ZDX) is to ensure the firewall and process allowlist
requirements are met so that any adverse impact on the end user experience is prevented or identified at
an early stage of deployment and tackled.
Although the ZDX client service does not make any forwarding decisions or change configurations on themachine, it is recommended to roll out the entitlement in a staggered fashion. Another reason to startwith pilot users is to address any environment-specific issues with a smaller rollout rather than finding theproblem at a larger scale.
1.1 Zscaler Client Connector version
Since ZDX runs as a module within ZCC, we have a one-to-many compatibility relationship. Ensuring
support for the latest ZDX version is key to take advantage of new features and bug fixes.
1.2 Pilot rollout
The rollout should begin with the users that are least resistant to change and self-sufficient in debugging
basic device issues such as Tier 3 helpdesk.
1.3 Successive staggered rollout
After a week of a pilot group of users running the service without disruptions, creating a staggered rollout
plan is next in deploying the entitlement across the entire user base.
It is recommended to start by rolling out across an entire team, such as the helpdesk, before continuingwith other teams. The number of rollout phases depends on the organization’s size and appetite forchange. The recommendation is to onboard up to 5,000 users at a time.

L3 HelpDesk

Rest of HelpDesk

Network & Operations-----------All Users

Sensitivity

Rollout 101:
1: Satisfy pre-requisites
2: Start small, pilot rollout3: Progressive rollout 		
across the population© 2023 Zscaler, Inc. All rights reserved.

White Paper

42. Application strategy

To ensure ZDX does not impact the end user experience that we are monitoring in the first place,the number of probes a user can run is limited to 30. This makes identifying which applications tomonitor crucial.
2.1 Usage-based
It is recommended to have applications selected based on the three below categories:

Criticalapplications

Group-specificcritical applications

ZscalerInternet AccessNoisy
applications

ZscalerPrivate AccessDirect2.1.1 Critical applications
These applications are critical from an employee/user productivity perspective and need to be monitored
across the board regularly. Common examples include the Microsoft 365 suite, Zoom, etc.
2.1.2 Group-specific critical applications
The group-specific category definition is creating groups of applications segmented based on a team’spriority. For example, the PagerDuty application would be critical for the Operations team, but theSalesforce application wouldn’t. Rather, the Sales team would prioritize the Salesforce application.

© 2023 Zscaler, Inc. All rights reserved.

White Paper5

2.1.3 Noisy applications
To derive maximum value from ZDX, monitoring unstable applications and generating the top ticket
volumes is essential.
The noisy application category is the only dynamic category, as the application state varies from stable tounstable at a given time. This means it’s important to revisit this list periodically.
During quarterly reviews, previously noisy applications can be disabled, and others onboarded. This also
leaves the opportunity to run a Deep Trace on disabled noisy applications should a user complain
about performance.

2.2 Delivery-based
At any given point in time, an organization relies on a multitude of applications. However, within thecontext of End User Experience Monitoring, it is not possible to measure performance of them all.
However, a select few delivery paths are utilized. With this in mind, if a good balance between thedelivery paths and usage-based app categories is achieved, existing data can be leveraged to baseline
performance based on forwarding used. For example, if your organization routes Outlook Online throughZscaler Internet Access (ZIA), that can be used to baseline performance for all apps funneled through ZIA.
2.2.1 Zscaler Internet Access (ZIA)• Data-driven SaaS applications are generally funneled through the ZIA Cloud Infrastructure.
2.2.2 Zscaler Private Access (ZPA)• Private applications that an organization manages and are not publicly accessible are primary
drivers for ZPA.
• Source IP compliance requirements can also drive apps to be routed through ZPA.
2.2.3. VPNs
• Organizations yet to adopt full zero trust will have private applications riding the legacy VPN tunnels.
2.2.4 Direct/Zscaler Bypassed• Real-time applications, such as UCaaS, are usually bypassed from ZIA.
• Users on the production network will usually route DIRECT to private applications.

© 2023 Zscaler, Inc. All rights reserved.

White Paper6

3. Application arrangement/services
Application layout in ZDX is essential in ensuring dashboards report the accurate status for aparticular service.

Recommended:
Single service per app

Application 1

Application 2

Application 3

Application 4

Not Recommended:
Multiple services bunchedunder a single ZDX app

Application 1

For instance, let’s say SAP is critical for my organization and there are two instances, Europe and Americas,
that are independent of each other and either can go down without the other. It’s then recommended toset up two applications: SAP Americas and SAP Europe.
Having both services combined under a single SAP application poses the risk of diluting trends for eachapplication and prevents the ability to understand how each instance is performing.

4. Probes
Correct probe setup is at the core of the whole experience monitoring exercise. The below sections
specifically describe each factor for the probe configuration.
4.1 Criteria: Run for select users/all
The 30 probe per user limit means utilization of active probes is critical to a platform’s health. Theseapproaches depend on the actual probe entitlement of an organization.

© 2023 Zscaler, Inc. All rights reserved.

White Paper

7

4.1.1 Organizations with 30 probes

HelpDeskOperations

Sales

Since you can only enable 30 probes on the platform at a time, running each probe from everyone withZDX enabled provides a more granular understanding of an application’s performance. However, if theapplication is not accessible to everyone, only users with access should be assigned to the probe.
4.1.2 Organizations with more than 30 probes

HelpDeskOperations

SalesHere, ensuring the entitled probes can be utilized is vital since the 30 probes per user limit is still enforced,
so proper planning is required. The specific group applications in this scenario should only be run for theconcerned group, as this helps ensure other users can run probes for maximum applications that are ofinterest to them.

© 2023 Zscaler, Inc. All rights reserved.

White Paper

8

4.2 Criteria: DDOS concerns

External apps

InternetInternal apps

PublicCloudSaaS

Datacenter

Zero TrustExchange

Web performance metrics from every endpoint

Work from anywhere, on any device

With ZDX, whenever a probe flows through Zscaler Clouds, the service edges perform smart caching toensure the end service does not get overwhelmed and, at the same time, gives users accurate
results back.
When the application is accessed directly by the client, all requests will reach the end service; hence, it’sessential to scale testing accordingly. Given that a privately-hosted application might not have the same
resources as a SaaS application, starting small with a group/department of geographically dispersed
users is recommended. The idea is to keep the total number of users small enough for the service tosustain requests. A Deep Trace can be run on demand if an issue arises for a user who is not running theapplication probing.

© 2023 Zscaler, Inc. All rights reserved.

White Paper

94.3 Web probe redirect/availability best practices
Setting the redirect and expected response codes is critical to ensure the metrics represent the desired
performance. The configuration should be based on how the application responds to an unauthenticated
HTTP request.
4.3.1 Application that responds with a 200 OK

GET https://zscaler.okta.com

200 OKHere, the success must be measured against the 200 and redirects followed.
4.3.2 Application that redirects to another service such as login with a 302 FoundGET https://app.box.com

302 Found https://account.box.com/login

In this scenario, measuring availability based on a 200 OK coming from a different service than the original
renders the metrics skewed. Hence, availability should be measured based on the ability of the original
application to redirect.
4.4 Cloud Path best practices

4.4.1 Protocol
Since the internet is “best-effort,” and devices on the internet do not consistently respond to a specificprotocol for network measurements, it’s recommended to use the “Adaptive” protocol for the Cloud
Path probe.

© 2023 Zscaler, Inc. All rights reserved.

White Paper

10In custom routing situations, such as a particular port/protocol being bypassed, it’s vital to use the specificprotocol and port to ensure correct Cloud Path probe metrics.
4.4.2 Packet Count
Packet Count is the number of traces each cloud path attempts, this plays a pivotal role on the number ofconnection requests a destination, perimeter firewalls, and intermediate devices receive. Setting this up to
an optimum level is critical for efficient probing.
Recommendation is to use a forwarding-based approach here, use 5 for SaaS ZIA//Direct destinations and3 for Private/ZPA destinations.
Note: Starting ZDX v3.3.1, packet count is auto-reduced to 3 for all probes when a trusted network is
detected to reduce impact. Deep Traces will continue to run with configured values.

5. Admin Access/RBAC
Provisioning sufficient and “just needed” levels of ZDX access to personnel is critical to maintaining
platform hygiene. Along with authentication, authorization is a major aspect of Administrator access.
ZDX provides the ability to granularly control the level of access an administrator has in the form of
permission-based roles.
These roles can then be assigned to administrators within a predefined scope.

Provisioning access to various different persona can hence be customized, below is an example:© 2023 Zscaler, Inc. All rights reserved.

White Paper11

5.1 Platform Owner
Full Read/Write access to the platform. The ZDX Super Admin built-in role should be assigned withorganization scope.
5.2 Helpdesk/Servicedesk
RW permission to Deep Trace, no permissions to Administrator and User management. RO permissions toprobe configuration and alerts with usernames visible and locations obfuscated.
5.3 Network Operations
RW permission to Deep Trace, no permissions to Administrator and User management. RO permissions toprobe configuration, alerts with usernames obfuscated and locations visible.
5.4 T1 Roles
This will be the most restricted role where an administrator is able to simply search for a user and look atperformance over the last 2-12 hours depending on the organization’s preference.
Here is the recommended RBAC Matrix:
Permissions

Platform Owner

Help Desk /Service Desk

Network Operations

T1 Service DeskZDX Dashboard

View Only

View Only

View Only

None

Application Overview

View Only

View Only

View Only

None

Application Dashboard

View Only

View Only

View Only

None

User Overview

View Only

View Only

View Only

None

User Dashboard

View Only

View Only

View Only

None

User Name

VisibleVisibleObfuscatedVisibleLocation

VisibleObfuscatedVisibleObfuscatedDevice Name

VisibleVisibleObfuscatedVisibleIP Address

VisibleVisibleVisibleVisibleCall Quality Configuration

Full

View Only

None

None

Call Quality Meetings

Full

View Only

None

None

Call Quality Applications

Full

View Only

None

None

Static Reports

Full

View Only

View Only

None

Configuration Access

Full

View Only

View Only

None

Administrator
Management

Full

None

None

None

User Management

Full

None

None

None

Dashboard Access

Device and User Information

UCaas Monitoring

Analytics© 2023 Zscaler, Inc. All rights reserved.

White Paper

12Permissions

Platform Owner

Help Desk /Service Desk

Network Operations

T1 Service DeskLocations

Full

None

View Only

None

Remote AssistanceManagement

Full

View Only

View Only

View Only

Deep Tracing

Full

Full

Full

FullAlerts

Full

View Only

View Only

NoneWebhooks

Full

View Only

View Only

None

Zscaler Client
Connector Portal

Full

None

None

None

Time Duration

Full

Full

Full

2-12 Hours

Inventory Management

Full

None

None

None

6. Alerting
ZDX collects many data points, but it’s not possible to manually sift through them. To address thischallenge, you should configure Alerts within ZDX. Alerts drive proactiveness with notifications that act asearly alarms before the wider population feels the impact.
The below sections describe best practices with different alert configurations:
6.1 Condition values
Setting alerts on the right conditions is critical to identifying issues before they take shape.
From an end user perspective, performance can be slow, or requests can fail at various stages. Below are
the quintessential metrics to be considered:Slowness

FailureComprehensive

Web Probe

Page Fetch, DNS Time

Availability

ZDX Score, Score Drops

Cloud Path Probe

Latency

Loss

CPU Utilization, Memory Utilization and Wi-Fi Signal Strength can similarly be configured from the device
health perspective.
6.2 Nesting
Nesting conditions under an alert rule is an important configuration. The above conditions nested logical‘OR’ will cover all the user experience issues.
At the same time, nesting them with an ‘AND’ won’t trigger the alert unless all the conditions are violated.
This can cause alerts to stay silent during an actual service degradation and is not recommended.

© 2023 Zscaler, Inc. All rights reserved.

White Paper13

6.3 Throttles: Violating Count, Devices, and Group BySetting additional throttles on top of violating conditions helps to get alert rules production ready.

Voilating Round Count

Impacted Devices

Alert Rule

Active Device

Group By

The four available throttling vectors are:6.3.1 Violating Rounds Count

6.3.3 Active Devices

Setting this to a number >1 is important toThis is very important to set a scalable global

prevent “spike” or a “one off” issue alerting, thealert. This should be set to the total population

alerts should rather trigger on sustained violationsof the smallest group by entity the organization

only, setting this at 2 violating counts is aneeds to be alerted upon.

right balance.

6.3.4 Group By

6.3.2 Impacted Devices

This value determines the scope for Devices andThis number should be set at the value which

Violating Count. The preferred value here varies

translates to a significant number of users

by Organization. An organization with known

seeing bad performance. In order to better

Zscaler locations available for the majority ofscale the alert, it’s recommended to set thisusers can use the same vs. others can utilize

to a percentage for violating devices and set aCities or Regions.

Minimum Device count to the smallest populationof the group by entity of interest.

© 2023 Zscaler, Inc. All rights reserved.

White Paper14

6.4 Tweaking/soak period
Alerts need constant tuning and tweaking over asoak period to be operationalized. At least a twoweek soak period is recommended after ZDX isrolled out to all users.

Sensitivity

Noise

Getting the right balance between alert noisiness
Alert Ruleand sensitivity is key to optimal alerting.
6.5 ITSM/IM integrations with webhook

To reap the full benefits of alerting, it’s recommended to both configure webhook notifications generating
automatic tickets and have a process built to mitigate tickets. This ensures a process is built around alerts
and incidents do not go unnoticed for long.

Digital Experience

Setting up webhooks to IM tools such as Slack and Microsoft Teams also helps get more eyes on issues
before they take bigger shape.

© 2023 Zscaler, Inc. All rights reserved.

White Paper

15

6.6 ITSM Routing
Once the tickets are created, the next step is routing them to the relevant team. Here, an approach needs
to be taken depending on the delivery path of the alerted application and the layer of alert rule.
These are our best practice recommendations and proposed handling:

App/Network Alert:
ZS forwarded

Network Alert:
ZS Bypassed

App/Network Alert:
Private App

Route to:Internal Zscaler teamRoute to:Internal Zscaler teamRoute to:Relevant Zscaler team

Next Steps

Next Steps

Next Steps• If issue in local
network/ISP: Forward
to internal network team

• Investigate theproblematic network
segment/node.

• Identify if its an
Application slowness
issue or a network issue
and mitigate accordingly• If issue in ZS network
segment: Open aZscaler support ticket“ZDX helped us identify the root cause of long-term
performance issues and our Helpdesk teams were able
to resolve user issues easily.”
Ballard Mattingly
Principal Cybersecurity Engineer, Liberty Mutual Insurance© 2023 Zscaler, Inc. All rights reserved.

White Paper16

7. Data analysis
7.1 Looking at metrics cohesively rather than subjectively
It’s important to look at metrics cohesively within the context of other available metrics (same timestamp)such as comparisons of application performance to network and device performance.
Here’s a good way of looking at Page Fetch time, Server Response Time, DNS Time and Latency to derive
actionable insights from the platform:• High page fetch
and server
response• High page fetch• High page fetch• High page fetch• No extra latency• No extra latency• No extra latency• No increase inserver responsetime• No increase inserver responsetime• High server
response time

• Increase in
DNS time• No increasein DNS

ServerSlowness

DNS Issue

Bandwidthfluctuation• Extra latency

NetworkIssue

7.2 Looking at surrounding data rounds, spike/pattern
When analyzing data, it’s important to have historical context on performance to understand where thebottlenecks lie.

On latency peaks, such as those demonstrated in the screenshot above, it’s vital to understand if a node is
injecting the extra latency each time.

© 2023 Zscaler, Inc. All rights reserved.

White Paper17

7.3 UCaaS monitoring
When it comes to Call Quality applications, ZDX Score is the most important metric on a user’s overall
meeting experience.
Looking at network performance, Jitter is an important factor that impacts real time
applications in particular.
With ZDX CLI View of Cloud Path, the StdDev column highlights the same, the highlighted hop in below
screenshot is adding between 14 to 140 ms of latency driving down the performance.

About Zscaler
Zscaler (NASDAQ: ZS) accelerates digital transformation so that customers can be more agile, efficient, resilient,and secure. The Zscaler Zero Trust Exchange protects thousands of customers from cyberattacks and data loss
by securely connecting users, devices, and applications in any location. Distributed across more than 150 data
centers globally, the SSE-based Zero Trust Exchange is the world’s largest inline cloud security platform. Learn
more at zscaler.com or follow us on Twitter @zscaler.
+1 408.533.0288

Zscaler, Inc. (HQ) • 120 Holger Way • San Jose, CA 95134© 2023 Zscaler, Inc. All rights reserved. Zscaler™,
Zero Trust Exchange™, Zscaler Internet Access™,ZIA™, Zscaler Private Access™, and ZPA™ areeither (i) registered trademarks or service marksor (ii) trademarks or service marks of Zscaler,Inc. in the United States and/or other countries.
Any other trademarks are the properties of theirrespective owners.

zscaler.com