Installation of RedHawk™ 7.5.3 Gold on the
NVIDIA® Jetson Xavier NX
Release Notes
October 9th, 2020
0898003-7.5.3-XavierNX

This page intentionally left blank1. Introduction
This document describes the process of installing the ARM64 version of RedHawk™ 7.5.3 on the
Jetson Xavier NX. The directions in this document supersede all others – they are specific toinstalling the software on Concurrent Real-Time’s RedHawk systems.

2. Requirements••••

Jetson Xavier NX with Internet access
A Micro SD card that is at least 32GB
An x86_64 host system running Ubuntu 16.04 or 18.04 with Internet access; note thatthe host system should be up-to-date with the latest available Ubuntu package updates
The RedHawk 7.5.3 for the Jetson Xavier NX optical media disk
NOTE
Make sure that /usr/bin/python exists on the Ubuntu host system;
install the python package if /usr/bin/python does not exist.

3. Installation
3.1.

Download SDK Manager onto the host system

On an x86_64 host system running Ubuntu 16.04 or 18.04, download version 1.3.1-7110 of SDK
Manager from the NVIDIA website via this URL:
https://developer.nvidia.com/jetpack
Click on the NVIDIA SDK Manager link to download the SDK Manager package (login may berequired). You should also click on the Jetson Xavier NX Developer Kit User Guide link to
download the User’s Guide PDF.
Place these downloaded files into a xavier directory in the home directory of the current user.

3.2.

Install the downloaded SDK Manager package

Open a terminal window and issue the following commands to install the SDK Manager:$ cd ~/xavier$ sudo apt install ./sdkmanager_1.3.1-7110_amd64.deb

3.3.

Use SDK Manager to flash the Xavier NX

Invoke the following command as the current user to begin using the SDK Manager:$ sdkmanager

The first action required by the SDK Manager is to log into the nvidia.com website using yourdeveloper or partner account email address and password. Enter the appropriate information andpress LOGIN to continue.

Upon login, you may immediately see the following dialog if you already have the Xavier NX
connected to the host system:

Select Jetson Xavier NX [developer kit version] as shown below:

Once selected, press OK to continue.
At this point, you can generally follow the instructions in the Jetson Xavier NX Developer Kit User
Guide along with the RedHawk-specific instructions detailed on the following pages.

1. On the STEP 01 - DEVELOPMENT ENVIRONMENT screen, click on the . . . icon in the Target
Hardware box to reveal the list of supported hardware configurations, as shown in thefollowing screenshot:

2. Click on the Jetson Xavier NX [developer kit version] (P3668-0000 module)
target hardware option to make it the selected target, as shown in the following
screenshot:3. Click on the . . . in the Linux box to reveal the list of supported JetPack versions, as
shown in the following screenshot:

4. Click on the JetPack 4.4 version to make it the selected version, as shown in thefollowing screenshot:5. Press Continue to advance to the STEP 02 – DETAILS AND LICENSE screen and click
I accept the terms and conditions of the license agreements at the bottom of
the window to accept the licenses, as shown in the following screenshot:

6. Press Continue to advance to the STEP 03 – SETUP PROCESS screen and SDK Manager
will begin downloading, flashing and installing software from NVIDIA, as shown below:

Depending on your network’s speed, the download can take 30 minutes or more, andcomplete flashing and installation can take 60 minutes or more. During the download andsetup, a detailed output log can be viewed under the TERMINAL tab.

7. Once the download and flashing has completed, you will be presented with a Flashdialog. Notice the Automatic setup pulldown menu as shown in the following dialog:

Click on Automatic setup and change the setting to Manual Setup as shown in thefollowing dialog:

Follow the steps listed on the Flash dialog to place the Jetson Xavier NX into recovery
mode and then click the Flash button to begin flashing.

NOTE
Issue the lsusb command on the host to verify that a single NVidia Corp.
entry with device ID 0955:7e19 is displayed; this is the best indication that the
Xavier NX has successfully entered recovery mode and is ready for flashing.

8. Midway through flashing you will need to connect the Xavier NX to a monitor, keyboard

and mouse to click through the System Configuration dialogs. Accept the license andthen choose values for all settings that are appropriate for your site.

Make sure to remember the username and password chosen because this information
will be needed by the SDK Manager in the following step. Press Continue to exit the
System Configuration dialogs and continue with flashing.

9. During flashing the SDK Manager will prompt you to fill in Username and Password
fields as shown in the following dialog.

Enter values for each field matching those entered into the System Configuration
dialog in the previous step and then click Install to continue flashing.

10. Once the flashing has completed you will automatically proceed to the STEP 04 –SUMMARY FINALIZATION screen.

Click Finish and Exit to exit the SDK Manager

3.4.

Expand root filesystem to utilize all Micro SD space

Due to a serious bug in SDK Manager, immediately after flashing the Xavier NX root filesystem
has insufficient free space, and this is true regardless of the actual size of the Micro SD card
being used. NVIDIA is aware of this problem, but they have not yet released a fix for it.
Before you can install RedHawk packages you must expand the root filesystem to utilize the fullspace that is available on the Micro SD card. To quickly resize the root filesystem, first copy thenx-sd-card-resize.sh script that is included on the RedHawk 7.5.3 for Jetson Xavier NX
optical disc to the Xavier NX and then invoke the script as follows:$ sudo ./nx-sd-card-resize.shThis script will run for several minutes and produce hundreds of lines of output, but eventually you
will see a message like the following, depending on the actual size of the Micro SD card:
The filesystem on /dev/mmcblk0p1 is now 7740178 (4k) blocks long.

At this point, reboot the Xavier NX and then verify that there is enough free space available in theroot filesystem. For example, with a 32GB Micro SD card you should see approximately 17GB of
free space, as shown below:$ df -h /Filesystem/dev/mmcblk0p1
$

Size
29G

Used Avail Use% Mounted on
12G
17G 42% /NOTE
If you do not expand the root filesystem before you attempt to install RedHawk
packages, RedHawk installation will fail when it fills up the Micro SD card andyou will need to go back to the start of this document to use SDK Manager again.

3.5.

Set NVIDIA Power Model to Maximum Performance

After SDK Manager has completed the Ubuntu installation on the Xavier NX, and while the Xavier
NX is still booted with the NVIDIA stock kernel, log into the Xavier NX using the username andpassword chosen during flashing and run the following commands:$ sudo nvpmodel -m 0
$ sudo nvpmodel -q

NOTE
You may see multiple pg warnings produced invoking theabove commands; these warnings can be ignored safely.
The second command invocation above should display the selected power mode as follows:
NV Fan Mode:quiet
NV Power Mode: MODE_15W_2CORE
0This confirms that the Xavier NX has been correctly configured for maximum performance.

3.6.

Copy RedHawk packages to the Xavier NX
NOTEThis section assumes you are using the RedHawk 7.5.3 for Jetson Xavier NX optical
media disc. Alternatively, if you wish to install RedHawk from the network refer toAppendix A, “Install RedHawk from Concurrent Real-Time Repositories.”
Once installed, continue with section 3.6 below to finalize the installation.

After SDK Manager has completed the Ubuntu installation on the Xavier NX, insert the RedHawk
7.5.3 for Jetson Xavier NX optical disc into the Ubuntu host’s optical drive and it should
automatically mount under the /media/$USER directory (e.g. /media/jane/RedHawk 7.5.3
aarch64).
Change to the mounted directory on the host and use networking to copy the RedHawk packages
into the newly installed Xavier NX, as illustrated by the following commands:$ cd /media/jane/RedHawk 7.5.3 aarch64/Packages$ scp *.deb user@host:/tmp

NOTE
Use the username, hostname and password chosen during flashing.

3.7.

Install RedHawk packages on the Xavier NX
NOTE
Before installing any RedHawk packages, make sure that there is enoughfree space available on the system. See section 3.4 above for more information.

To install the RedHawk packages, log into the Xavier NX (either via ssh or via the graphicalconsole) and issue the following commands:$ cd /tmp$ sudo apt install ./ccur*.deb

Enter the password for the current user if the sudo command prompts for it.
NOTE
Several warnings may be displayed during package installation, including:
•
•
•
•
•
•

cryptsetup: WARNING: could not determine root device…
Warning; /sbin/fsck.rootfs doesn't exist, can't install to…
N: Can’t drop privileges for downloading as file…
insserv: warning: script nv-run-date, nv-run-once-install, …
Warning: couldn’t identify filesystem type for fsck hook…
/sbin/ldconfig.real: Warning: ignoring configuration file…

You can safely ignore these warnings as they are not relevant
to RedHawk 7.5.3 on Jetson Xavier NX embedded installations.
At this point all the RedHawk packages, including the RedHawk kernels, should have installedsuccessfully on the Xavier NX. However, additional steps in the next section are required before
you can boot into the RedHawk kernels.

Add RedHawk entries to the extlinux.conf file

3.8.

Perform the following steps on the Jetson to create boot entries for the RedHawk kernels:
1.

Create a backup of the /boot/extlinux/extlinux.conf file.
$ cd /boot/extlinux$ sudo cp extlinux.conf extlinux.bak

2.

Open the extlinux.conf file in your preferred editor. For example, if using vi:$ sudo vi extlinux.conf

3.

Duplicate all the primary kernel entry text lines to create a second identical entry
below the initial entry. For example:
TIMEOUT 30
DEFAULT primary
MENU TITLE L4T boot options
LABEL primary
MENU LABEL primary kernel
LINUX /boot/ImageINITRD /boot/initrd
APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait ...
LABEL primary
MENU LABEL primary kernel
LINUX /boot/ImageINITRD /boot/initrd
APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait ...

4.

Customize the first kernel entry to point to the desired RedHawk kernel. For example, tocreate an entry for the RedHawk trace kernel, modify the lines as follows:
LABEL redhawk-trace
MENU LABEL redhawk-trace kernel
LINUX /boot/Image-4.9.140-rt94-r32.4.3-tegra-RedHawk-7.5.3-trace
INITRD /boot/initrd.img-4.9.140-rt94-r32.4.3-tegra-RedHawk-7.5.3-trace
APPEND ${cbootargs} quiet root=/dev/mmcblk0p1 rw rootwait ...

NOTE
Modifying the first kernel entry will allow the NVIDIA kernel to be booted asa fallback if any typos were entered during the customization of the first entry.
Failure to follow this guideline can render the Xavier NX unbootable andrequire re-flashing if a typo is accidentally introduced during editing.
5. Change the default kernel to the newly added RedHawk kernel by changing the DEFAULT
entry at the top of the file:
TIMEOUT 30
DEFAULT redhawk-trace

Note that if your Xavier NX has been set up with a serial console, you may choose to skip
step 5 and instead interactively decide which kernel you wish to boot each time thesystem restarts.

3.9.

Verify RedHawk kernel installation

The Xavier NX should now be running the selected RedHawk kernel. To verify this, issue thefollowing command:$ uname -r

You should see output displayed like the following:
4.9.140-rt94-r32.4.3-tegra-RedHawk-7.5.3-trace

Please contact Concurrent Real-Time technical support if you had any problems during thisinstallation (support@concurrent-rt.com or 1-800-245-6453).

4. Known Issues
Special consideration should be given to the following areas.

4.1.

Xavier NX hardware not optimized for real-time at bootBy default, at boot time the Xavier NX hardware is optimized for maximum throughput instead of
maximum real-time performance; in this mode real-time applications will experience lessdeterminism and higher latencies during execution.
To run real-time workloads on the Xavier NX with high determinism and significantly lower
latencies, RedHawk provides a new /proc/rt_safe_mask control file to give users the ability
to select the level of real-time performance on the Xavier NX. For example, invoking the following
command as the root user will maximize real-time performance:# echo 63 > /proc/rt_safe_mask
The above command enables maximum real-time performance on all 6 CPU cores on the Xavier
NX, and this is required to achieve the maximum real-time performance of the whole system.
Note that 63 is a bit mask (i.e. hex 0x3F or binary 0b111111) indicating that each of the 6
individual CPU cores is to be configured for maximum real-time performance.

NOTE
Assigning values to /proc/rt_safe_mask that are greater than 63
can cause the Xavier NX to become unstable and may crash the system.

Users may choose to exclude one or more CPU cores from the bit mask to sacrifice a degree ofreal-time performance in exchange for improved throughput; for example, invoking the following
command as the root user excludes CPU core 0 from the bit mask:# echo 62 > /proc/rt_safe_mask
The above command will still improve real-time performance compared to the default setting of 0
(where no CPU cores are optimized for real-time) but the improvement will not be as good as
when all CPU cores are enabled for real-time. However, in this mode throughput will improve.
Finally, users can also return to the default maximum throughput state of the Xavier NX (where
no CPU cores are optimized for real-time) by invoking the following command as the root user:# echo 0 > /proc/rt_safe_mask

NOTE
Because the Xavier NX defaults to optimizing for maximum throughput, it may be advantageous
to start real-time applications and let them run for some amount of time before setting/proc/rt_safe_mask to 63 to maximize real-time performance; users are encouraged
to experiment with this technique to determine whether any advantages may be gained.

4.2.

Sibling cores can interfere with shielding

Xavier NX cores utilize a 3-tier cache as shown in the following diagram:

Each core has its own L1 cache, but the L2 caches are always shared between two sibling cores.
In the above diagram, core 0 and core 1 are siblings, core 2 and core 3 are siblings, and core 4
and core 5 are siblings.
When shielding a core on the Xavier NX for a real-time application, activity on the shielded core’s
sibling core can modify the shared L2 cache and impact the determinism and performance of thereal-time application. In order to achieve maximum real-time performance, both sibling cores
must be shielded, with the unused sibling core kept idle.

4.3.

Entropy gathering daemon interferes with real-time

Ubuntu 18.04 enables a new Hardware Volatile Entropy Gathering daemon (haveged) that
continuously samples various hardware resources (e.g. cache, branch predictions, memorytranslation tables) to collect entropy and improve random number generation.
This daemon’s constant hardware probing activity can negatively impact real-time performanceeven on shielded CPU cores, and the daemon should be stopped before starting real-timeworkloads by invoking the following command as the root user:# systemctl stop haveged.service
Note that disabling the entropy gathering service is not recommended because some Ubuntu
startup services require a minimum level of entropy to be gathered before they will start; thisincludes services required for the X Server to properly initialize. However, after the Xavier NX has
started with graphics displaying you can safely stop the service to improve real-time performance.

4.4.

RedHawk kernels and OEM verification failures

Booting any of the RedHawk kernels on the Xavier NX will result in the following warnings being
displayed almost immediately after the boot process begins:[0014.855] I> Validate kernel ...
[0014.855] I> T19x: Authenticate kernel (bin_type: 37), max size 0x5000000[0014.856] E> Stage2Signature validation failed with SHA2!!
[0014.856] C> OEM authentication of kernel header failed!
[0014.857] W> Failed to validate kernel binary (err=1077936152)[0014.857] W> Security fuse not burned, continue to bootThese warnings can be safely ignored because OEM security fuses are not burned by default on
Xavier NX hardware platforms. For more information, refer to the NVIDIA Jetson Xavier NX Fuse
Specification available on the NVIDIA Developer website.

4.5.

Lockdep is turned off shortly after bootThe kernel code provided by NVIDIA has not been optimized for use with debug kernels, and thisis most evident when the kernel’s CONFIG_LOCKDEP_SUPPORT feature is enabled. Because
RedHawk debug kernels (debug and prt-debug) are built with this feature, lockdep
informational messages will occasionally appear in the system’s message journal. For example:
INFO: possible recursive locking detected...
INFO: lockdep is turned off
These messages are solely informational, and they do not appear to cause any real-time or
behavioral problems with the system. Concurrent is working with NVIDIA to address thesewarnings and hopefully they will be eliminated in future NVIDIA kernels.

4.6.

Kernel debugger not accessible via serial console

The KGDB kernel debugger for the Xavier NX is currently unable to communicate over the serial
console interface (/dev/ttyTCU0). Concurrent is working with NVIDIA to address this issue
and hopefully it will be corrected in future NVIDIA kernels.

4.7.

Warnings accessing /dev/mem

Attempting to directly access /dev/mem on the Xavier NX is not permitted and will result inwarnings output to the system console. For example, invoking the dmidecode command while a
RedHawk kernel is running will output an error like the following:
Program dmidecode tried to access /dev/mem
These messages are informational and can be safely ignored.
Note that if the stock NVIDIA kernel is running, issuing dmidecode will crash the system andcause it to automatically reboot.

4.8.

Change to PAM Capability module directory

The PAM Capability module directory that is documented in some versions of the RedHawk
User’s Guide is incorrect for the Xavier NX. The correct PAM Capability module location should
be shown as:/lib/aarch64-linux-gnu/security/pam_capability.so
If you wish to use the PAM Capability feature with services in the /etc/pam.d directory, thenyou should refer to the PAM Capability module as follows:
session required /lib/aarch64-linux-gnu/security/pam_capability.so

4.9.

Erroneous boot-time warnings

Booting the stock NVIDIA kernel or the RedHawk Linux kernels on the Xavier NX will produce
several warnings, including the following:
WARNING: at platform/driers/pg/pg/pg-cv-t194.c:73
WARNING: clk_get_rate: clk_power_ungate on gated domain 27 for gpcclk
WARNING: pll_d2 has no dyn ramp

These warnings have been reported to NVIDIA but there is no current resolution. The warnings
do not appear to impact the real-time performance of the RedHawk and can be safely ignored.

A. Install RedHawk from Concurrent Real-Time Repositories
RedHawk can be installed over the network utilizing the Concurrent Real-Time software repositories.
This has the advantage that it will install the latest updates that have been released, however some
up-front setup is required to perform the installation; follow the steps below to complete this setup.

A.1.

Install Concurrent Real-Time Public Keys

Concurrent Real-Time signs its APT repositories with a GPG-generated DSA and ElGamal key
pair. You need to install Concurrent Real-Time's public key so that the APT system can authorize
usage of the repositories.
You can download the ccur-public-keys file and import the key file as needed into APT's key ring.
You should import the key file before attempting to access software from Concurrent Real-Time's
repositories.
The following commands will download the public key and install it.
wget -q http://redhawk.concurrent-rt.com/network/ccur-public-keys
apt-key add ccur-public-keys
If you wish to check the authenticity of the public keys file you downloaded from Concurrent RealTime's web site, request the key fingerprints from Concurrent Software Support and comparethem to the fingerprints of the keys you downloaded.
You can obtain the fingerprints of the keys you downloaded with the following command:
gpg --with-fingerprint ccur-public-keys

A.2.

Install Repository Definition Files

Repository definition files tell the APT installation and update system how to access specificrepositories. The definition files must reside in the /etc/apt/sources.list.d/ directory andthe file name must end .list; e.g. /etc/apt/sources.list.d/ccur.list.
Use the following format for the repository definition of the RedHawk 7.5.3 for Jetson Xavier NX
product:
deb http://redhawk.concurrent-rt.com/ubuntu/login/passwd/redhawk/nx-xavier 7.5.n core

For example, the following entry substitutes a fictional login and password:
deb http://redhawk.concurrent-rt.com/ubuntu/LR12345/t0ps3cr3t/redhawk/nx-xavier 7.5.n core

Note that if the NightStar tools have been purchased for this machine, then you should also add arepository definition for the NightStar tools below the previously added line. For example:
deb http://redhawk.concurrent-rt.com/ubuntu/LR12345/t0ps3cr3t/nightstar 4.6 rt

Once the repository definition file is in place the repository setup is complete. Run the following
command as root to ensure that the software repositories have been added correctly:
sudo apt update
If you see errors or warnings produced that reference the newly added ccur.list file, thenverify that all the above steps have all been completed as documented.

A.3.

Install RedHawk Packages

Now that the setup is done, simply issue the following command to install all the latest RedHawk
7.5.3 packages (and NightStar 4.6 packages if purchased) onto the current system:
apt install 'ccur-*'Once this command completes the RedHawk packages are installed. Continue with Section 3.5,
“Flash desired RedHawk kernel using bootburn.sh," in order to finalize the RedHawk installation.

A.4.

SupportIf you need assistance, please contact the Concurrent Real-Time Software Support Center at our
toll-free number 1-800-245-6453. For calls outside the continental United States, the number is 1954-283-1822. The Software Support Center operates Monday through Friday from 8 a.m. to 5
p.m., Eastern Standard Time.
You may also submit a request for assistance at any time by using the Concurrent Real-Time
website at http://concurrent-rt.com/support or by sending an email to support@concurrent-rt.com.