IBM How to process an IBM SDK core dump with Jextract (AIX, Linux, Windows) - United States crash; coredump; dumps; dump; dmp; dmps; jextracted; jextract; userdump; dtfj TECHNOTE (TROUBLESHOOTING)

PROBLEM(ABSTRACT)
 How do you process a raw core dump using Jextract? 

Jextract is a tool provided with the IBM SDK to process and compress a core dump for later processing with tools such as DTFJ, which can be downloaded with IBM Support Assistant. 

RESOLVING THE PROBLEM
You can process a core dump with Jextract, as long as the dump itself was produced by the IBM WebSphere Application Server JVM and is processed on the same system as the dump. 

A processed dump can be used in both the Eclipse Memory Analyzer Tool [http://www.ibm.com/developerworks/java/jdk/tools/mat.html] and the DTFJ Dump Analyzer [http://www.ibm.com/developerworks/java/jdk/tools/dumpanalyzer/]. Both are included as downloadable tools from within IBM Support Assistant Workbench [http://www.ibm.com/software/support/isa/]. 




Locating the Core Dump 

For WebSphere Application Server installations, the default location for the core dumps is in the profile's root directory: <WAS_HOME>/profiles/<PROFILE_NAME>/ 

The file extension will usually be *.dmp, but on some versions of AIX and Linux, the file may simply be named core. The process id and the date stamp are normally appended to the name of the file. 




Processing the Dump 

Execute the jextract command on the same system and installation of WebSphere Application Server that the core dump was produced on. If you don't use the full path on AIX or Linux, make sure to use ./jextract (including the dot-slash) so the correct executable is picked up. 

Replace [COREDUMP_PATH] with the full path plus filename of the core dump produced. 

 

If you do not know if compressed references is enabled, scroll to the FAQ section for a brief explanation. 

 

Determining the Java Directory 

Because different profiles on WebSphere Application Server 8.5.5.X and later can use different SDKs, the java directory inside the installation folder can differ. To determine the path, check the process list for the application server process and see which folder the java executable is being launched from. 

 

NOTE: If you are on WebSphere Liberty and are using an IBM SDK, the entire path to that SDK will differ. 

 




32-bit JVM and
64-bit JVM with Compressed References disabled
AIX/Linux: 
<WAS_HOME>/<JAVA_DIR>/jre/bin/jextract [COREDUMP_PATH]

Windows: 
<WAS_HOME>\<JAVA_DIR>\jre\bin\jextract.exe [COREDUMP_PATH]




64-bit JVM with Compressed References enabled
Does not apply to WebSphere Application Server 6.1
AIX/Linux: 
<WAS_HOME>/<JAVA_DIR>/jre/bin/jextract -J-Xcompressedrefs [COREDUMP_PATH]

Windows: 
<WAS_HOME>\<JAVA_DIR>\jre\bin\jextract.exe -J-Xcompressedrefs [COREDUMP_PATH] 

 




Max Heap Size 

To adjust the max heap size, you can add -J-Xmx to adjust the size. For example, -J-Xmx1024m allows for a max heap size of 1024 MB when jextract is run. 

 

 




Files Generated 

The output will determine where the compressed data is stored (usually a .zip extension, .sdff is seen on older environments). Check the location of the core dump as that will be the most common location. 

Core files are usually compressible since a lot of data is zero; 10x compression can be expected in the resulting zip file 

 

If you find XML files in the output directory, this may mean jextract failed, as these files are supposed to be placed automatically within the SDFF or ZIP file. 

 




Error Messages 

EyeCatcher Error
Unable to find J9VMRAS eyecatcher
Error. Internal failure detected during jextract 

This error message indicates that jextract failed to identify the dump. This could be due to a number of things, including 

* The dump was not produced by WebSphere Application Server
* The dump was produced from a native library or component such as a JNI component or database driver.
* (AIX only) The dump generated is in an incompatible dump format due to an AIX OS issue (see APAR IZ88097 [http://www.ibm.com/support/docview.wss?uid=swg1IZ88097] for more details). 

 

Out of Memory Error 

Sometimes the core dump is much larger than the default heap size can manage, and the recommendation is to increase the heap size using the -J-Xmx option. 

 




FAQ 

How do I determine if compressed references is enabled? 

Compressed references were first introduced in 64-bit builds of IBM SDK 6.0. If WebSphere Application Server is at version 7.0.0.3 or higher, it will be enabled by default; see APAR PK75174 [http://www.ibm.com/support/docview.wss?uid=swg1PK75174]. 

Checking the JVM arguments in the admin console can verify if compressed references was enabled manually. However, it's best to check the ps output or the javacore for the argument -Xcompressedrefs 

If you find the argument -Xnocompressedrefs has been added, this means it's disabled (see page 7 of this document [ftp://public.dhe.ibm.com/software/webserver/appserv/was/WAS_V7_64-bit_performance.pdf] for more info)) 

 

What happens to jextract if I cannot determine if compressed references was enabled? 

If you run jextract and did not realize compressed references was enabled, the output will ask you to rerun jextract and add the following argument: -J-Xcompressedrefs 



RELATED INFORMATION
 MustGather: Crash on AIX [http://www.ibm.com/support/docview.wss?uid=swg21055387]
MustGather: Crash on Linux [http://www.ibm.com/support/docview.wss?uid=swg21104706]
MustGather: Crash on Windows [http://www.ibm.com/support/docview.wss?uid=swg21053924]
Generating Userdumps on Windows [http://www.ibm.com/support/docview.wss?uid=swg21138203]
Crash on AIX produces no core or a truncated core [http://www.ibm.com/support/docview.wss?uid=swg21052642]
Crash on Linux produces no core or a truncated core [http://www.ibm.com/support/docview.wss?uid=swg21115658]

Question: How to proceed when jextract utility is throwing an OutOfMemory error ? 

I was getting a crash and was told to provide a "jextracted core dump". I ran the jextract command as instructed:

/java/jre/bin/jextract [CORE_PATH]

but I am getting now an OutOfMemory error when jextract itself is running so I cannot proceed with the original crash investigation.

Answer: Sometimes the core dump is much larger than the default heap size can manage, and the recommendation is to increase the heap size using the -J-Xmx option.