Package: nvitop
nvitopAn interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management. The full API references host athttps://nvitop.readthedocs.io.Monitor mode ofnvitop.(TERM: GNOME Terminal / OS: Ubuntu 16.04 LTS (over SSH) / Locale:en_US.UTF-8)Table of ContentsFeaturesRequirementsInstallationUsageDevice and Process StatusResource MonitorFor Docker UsersFor SSH UsersCommand Line Options and Environment VariablesKeybindings for Monitor ModeCUDA Visible Devices Selection ToolCallback Functions for Machine Learning FrameworksCallback for TensorFlow (Keras)Callback for PyTorch LightningTensorBoard IntegrationMore than a MonitorQuick StartStatus SnapshotResource Metric CollectorLow-level APIsDeviceProcessHost (inherited from psutil)ScreenshotsChangelogLicenseCopyright Noticenvitopis an interactive NVIDIA device and process monitoring tool. It has a colorful and informative interface that continuously updates the status of the devices and processes. As a resource monitor, it includes many features and options, such as tree-view, environment variable viewing, process filtering, process metrics monitoring, etc. Beyond that, the package also ships aCUDA device selection toolnviselfor deep learning researchers. It also provides handy APIs that allow developers to write their own monitoring tools. Please refer to sectionMore than a Monitorand the full API references athttps://nvitop.readthedocs.iofor more information.Process filtering and a more colorful interface.Compare tonvidia-smi.FeaturesInformative and fancy output: show more information thannvidia-smiwith colorized fancy box drawing.Monitor mode: can run as a resource monitor, rather than print the results only once.bar charts and history graphsprocess sortingprocess filteringsend signals to processes with a keystroketree-view screen for GPU processes and their parent processesenvironment variable screenhelp screenmouse supportInteractive: responsive for user input (from keyboard and/or mouse) in monitor mode. (vs.gpustat&py3nvml)Efficient:query device status usingNVML Python bindingsdirectly, instead of parsing the output ofnvidia-smi. (vs.nvidia-htop)support sparse query and cache results withTTLCachefromcachetools. (vs.gpustat)display information using thecurseslibrary rather thanprintwith ANSI escape codes. (vs.py3nvml)asynchronously gather information using multi-threading and correspond to user input much faster. (vs.nvtop)Portable: work on both Linux and Windows.get host process information using the cross-platform librarypsutilinstead of callingps -p <pid>in a subprocess. (vs.nvidia-htop&py3nvml)written in pure Python, easy to install withpip. (vs.nvtop)Integrable: easy to integrate into other applications, more than monitoring. (vs.nvidia-htop&nvtop)nvitopsupports Windows!(SHELL: PowerShell / TERM: Windows Terminal / OS: Windows 10 / Locale:en-US)RequirementsPython 3.7+NVIDIA Management Library (NVML)nvidia-ml-pypsutilcachetoolstermcolorcurses*(withlibncursesw)NOTE:TheNVIDIA Management Library (NVML)is a C-based programmatic interface for monitoring and managing various states. The runtime version of the NVML library ships with the NVIDIA display driver (available atDownload Drivers | NVIDIA), or can be downloaded as part of the NVIDIA CUDA Toolkit (available atCUDA Toolkit | NVIDIA Developer). The lists of OS platforms and NVIDIA-GPUs supported by the NVML library can be found in theNVML API Reference.This repository contains a Bash script to install/upgrade the NVIDIA drivers for Ubuntu Linux. For example:gitclone--depth=1https://github.com/XuehaiPan/nvitop.git&&cdnvitop# Change to tty3 console (required for desktop users with GUI (tty2))# Optional for SSH userssudochvt3# or use keyboard shortcut: Ctrl-LeftAlt-F3bashinstall-nvidia-driver.sh--package=nvidia-driver-470# install the R470 driver from ppa:graphics-driversbashinstall-nvidia-driver.sh--latest# install the latest driver from ppa:graphics-driversNVIDIA driver installer for Ubuntu Linux.Runbash install-nvidia-driver.sh --helpfor more information.*Thecurseslibrary is a built-in module of Python on Unix-like systems, and it is supported by a third-party package calledwindows-curseson Windows using PDCurses. Inconsistent behavior ofnvitopmay occur on different terminal emulators on Windows, such as missing mouse support.InstallationIt is highly recommended to installnvitopin an isolated virtual environment.Simple installation and run viapipx:pipxrunnvitopInstall from PyPI ():pip3install--upgradenvitopInstall from conda-forge ():condainstall-cconda-forgenvitopInstall the latest version from GitHub ():pip3install--upgradepipsetuptools
pip3installgit+https://github.com/XuehaiPan/nvitop.git#egg=nvitopOr, clone this repo and install manually:gitclone--depth=1https://github.com/XuehaiPan/nvitop.gitcdnvitop
pip3install.NOTE:If you encounter the"nvitop: command not found"error after installation, please check whether you have added the Python console script path (e.g.,"${HOME}/.local/bin") to yourPATHenvironment variable. Alternatively, you can usepython3 -m nvitop.MIG Device Support.UsageDevice and Process StatusQuery the device and process status. The output is similar tonvidia-smi, but has been enriched and colorized.# Query the status of all devices$nvitop-1# or use `python3 -m nvitop -1`# Specify query devices (by integer indices)$nvitop-1-o01# only show <GPU 0> and <GPU 1># Only show devices in `CUDA_VISIBLE_DEVICES` (by integer indices or UUID strings)$nvitop-1-ov# Only show GPU processes with the compute context (type: 'C' or 'C+G')$nvitop-1-cWhen the-1switch is on, the result will be displayedONLY ONCE(same as the default behavior ofnvidia-smi). This is much faster and has lower resource usage. SeeCommand Line Optionsfor more command options.There is also a CLI tool callednviselthat ships with thenvitopPyPI package. SeeCUDA Visible Devices Selection Toolfor more information.Resource MonitorRun as a resource monitor:# Monitor mode (when the display mode is omitted, `NVITOP_MONITOR_MODE` will be used)$nvitop# or use `python3 -m nvitop`# Automatically configure the display mode according to the terminal size$nvitop-mauto# shortcut: `a` key# Arbitrarily display as `full` mode$nvitop-mfull# shortcut: `f` key# Arbitrarily display as `compact` mode$nvitop-mcompact# shortcut: `c` key# Specify query devices (by integer indices)$nvitop-o01# only show <GPU 0> and <GPU 1># Only show devices in `CUDA_VISIBLE_DEVICES` (by integer indices or UUID strings)$nvitop-ov# Only show GPU processes with the compute context (type: 'C' or 'C+G')$nvitop-c# Use ASCII characters only$nvitop-U# useful for terminals without Unicode support# For light terminals$nvitop--light# For spectrum-like bar charts (requires the terminal supports 256-color)$nvitop--colorfulYou can configure the default monitor mode with theNVITOP_MONITOR_MODEenvironment variable (defaultautoif not set). SeeCommand Line Options and Environment Variablesfor more command options.In monitor mode, you can useCtrl-c/T/Kkeys to interrupt / terminate / kill a process. And it's recommended toterminateorkilla process in thetree-view screen(shortcut:t). For normal users,nvitopwill shallow other users' processes (in low-intensity colors). Forsystem administrators, you can usesudo nvitopto terminate other users' processes.Also, to enter the process metrics screen, select a process and then press theEnter/Returnkey .nvitopdynamically displays the process metrics with live graphs.Watch metrics for a specific process (shortcut:Enter/Return).Presshfor help orqto return to the terminal. SeeKeybindings for Monitor Modefor more shortcuts.nvitopcomes with a help screen (shortcut:h).For Docker UsersBuild and run the Docker image usingnvidia-docker:gitclone--depth=1https://github.com/XuehaiPan/nvitop.git&&cdnvitop# clone this repo firstdockerbuild--tagnvitop:latest.# build the Docker imagedockerrun-it--rm--runtime=nvidia--gpus=all--pid=hostnvitop:latest# run the Docker containerTheDockerfilehas an optional build argumentbasetag(default:450-signed-ubuntu22.04) for the tag of imagenvcr.io/nvidia/driver.NOTE:Don't forget to add the--pid=hostoption when running the container.For SSH UsersRunnvitopdirectly on the SSH session instead of a login shell:sshuser@host-tnvitop# installed by `sudo pip3 install ...`sshuser@host-t'~/.local/bin/nvitop'# installed by `pip3 install --user ...`NOTE:Users need to add the-toption to allocate a pseudo-terminal over the SSH session for monitor mode.Command Line Options and Environment VariablesTypenvitop --helpfor more command options:usage: nvitop [--help] [--version] [--once | --monitor [{auto,full,compact}]]
              [--interval SEC] [--ascii] [--colorful] [--force-color] [--light]
              [--gpu-util-thresh th1 th2] [--mem-util-thresh th1 th2]
              [--only idx [idx ...]] [--only-visible]
              [--compute] [--only-compute] [--graphics] [--only-graphics]
              [--user [USERNAME ...]] [--pid PID [PID ...]]

An interactive NVIDIA-GPU process viewer.

options:
  --help, -h            Show this help message and exit.
  --version, -V         Show nvitop's version number and exit.
  --once, -1            Report query data only once.
  --monitor [{auto,full,compact}], -m [{auto,full,compact}]
                        Run as a resource monitor. Continuously report query data and handle user inputs.
                        If the argument is omitted, the value from `NVITOP_MONITOR_MODE` will be used.
                        (default fallback mode: auto)
  --interval SEC        Process status update interval in seconds. (default: 2)
  --ascii, --no-unicode, -U
                        Use ASCII characters only, which is useful for terminals without Unicode support.

coloring:
  --colorful            Use gradient colors to get spectrum-like bar charts. This option is only available
                        when the terminal supports 256 colors. You may need to set environment variable
                        `TERM="xterm-256color"`. Note that the terminal multiplexer, such as `tmux`, may
                        override the `TREM` variable.
  --force-color         Force colorize even when `stdout` is not a TTY terminal.
  --light               Tweak visual results for light theme terminals in monitor mode.
                        Set variable `NVITOP_MONITOR_MODE="light"` on light terminals for convenience.
  --gpu-util-thresh th1 th2
                        Thresholds of GPU utilization to determine the load intensity.
                        Coloring rules: light < th1 % <= moderate < th2 % <= heavy.
                        ( 1 <= th1 < th2 <= 99, defaults: 10 75 )
  --mem-util-thresh th1 th2
                        Thresholds of GPU memory percent to determine the load intensity.
                        Coloring rules: light < th1 % <= moderate < th2 % <= heavy.
                        ( 1 <= th1 < th2 <= 99, defaults: 10 80 )

device filtering:
  --only idx [idx ...], -o idx [idx ...]
                        Only show the specified devices, suppress option `--only-visible`.
  --only-visible, -ov   Only show devices in the `CUDA_VISIBLE_DEVICES` environment variable.

process filtering:
  --compute, -c         Only show GPU processes with the compute context. (type: 'C' or 'C+G')
  --only-compute, -C    Only show GPU processes exactly with the compute context. (type: 'C' only)
  --graphics, -g        Only show GPU processes with the graphics context. (type: 'G' or 'C+G')
  --only-graphics, -G   Only show GPU processes exactly with the graphics context. (type: 'G' only)
  --user [USERNAME ...], -u [USERNAME ...]
                        Only show processes of the given users (or `$USER` for no argument).
  --pid PID [PID ...], -p PID [PID ...]
                        Only show processes of the given PIDs.nvitopcan accept the following environment variables for monitor mode:NameDescriptionValid ValuesDefault ValueNVITOP_MONITOR_MODEThe default display mode (a comma-separated string)auto/full/compactplain/colorfuldark/lightauto,plain,darkNVITOP_GPU_UTILIZATION_THRESHOLDSThresholds of GPU utilization10,75,1,99, ...10,75NVITOP_MEMORY_UTILIZATION_THRESHOLDSThresholds of GPU memory percent10,80,1,99, ...10,80LOGLEVELLog level for log messagesDEBUG,INFO,WARNING, ...WARNINGFor example:# Replace the following export statements if you are not using Bash / ZshexportNVITOP_MONITOR_MODE="full,light"# Full monitor mode with light terminal tweaksnvitopFor convenience, you can add these environment variables to your shell startup file, e.g.:# For Bashecho'export NVITOP_MONITOR_MODE="full"'>>~/.bashrc# For Zshecho'export NVITOP_MONITOR_MODE="full"'>>~/.zshrc# For Fishecho'set -gx NVITOP_MONITOR_MODE "full"'>>~/.config/fish/config.fish# For PowerShell'$Env:NVITOP_MONITOR_MODE = "full"'>>$PROFILE.CurrentUserAllHostsKeybindings for Monitor ModeKeyBindingqQuit and return to the terminal.h/?Go to the help screen.a/f/cChange the display mode toauto/full/compact.r/<C-r>/<F5>Force refresh the window.<Up>/<Down><A-k>/<A-j><Tab>/<S-Tab><Wheel>Select and highlight a process.<Left>/<Right><A-h>/<A-l><S-Wheel>Scroll the host information of processes.<Home>Select the first process.<End>Select the last process.<C-a>^Scroll left to the beginning of the process entry (i.e. beginning of line).<C-e>$Scroll right to the end of the process entry (i.e. end of line).<PageUp>/<PageDown><A-K>/<A-J>[/]scroll entire screen (for large amounts of processes).<Space>Tag/untag current process.<Esc>Clear process selection.<C-c>ISendsignal.SIGINTto the selected process (interrupt).TSendsignal.SIGTERMto the selected process (terminate).KSendsignal.SIGKILLto the selected process (kill).eShow process environment.tToggle tree-view screen.<Enter>Show process metrics.,/.Select the sort column./Reverse the sort order.on(oN)Sort processes in the natural order, i.e., in ascending (descending) order ofGPU.ou(oU)Sort processes byUSERin ascending (descending) order.op(oP)Sort processes byPIDin descending (ascending) order.og(oG)Sort processes byGPU-MEMin descending (ascending) order.os(oS)Sort processes by%SMin descending (ascending) order.oc(oC)Sort processes by%CPUin descending (ascending) order.om(oM)Sort processes by%MEMin descending (ascending) order.ot(oT)Sort processes byTIMEin descending (ascending) order.HINT:It's recommended to terminate or kill a process in the tree-view screen (shortcut:t).CUDA Visible Devices Selection ToolAutomatically selectCUDA_VISIBLE_DEVICESfrom the given criteria. Example usage of the CLI tool:#Alldevicesbutsorted$nvisel# or use `python3 -m nvitop.select`6,5,4,3,2,1,0,7,8#Asimpleexampletoselect4devices$nvisel-n4# or use `python3 -m nvitop.select -n 4`6,5,4,3#Selectavailabledevicesthatsatisfythegivenconstraints$nvisel--min-count2--max-count3--min-free-memory5GiB--max-gpu-utilization606,5,4#Set`CUDA_VISIBLE_DEVICES`environmentvariableusing`nvisel`$exportCUDA_DEVICE_ORDER="PCI_BUS_ID"CUDA_VISIBLE_DEVICES="$(nvisel-c1-f10GiB)"CUDA_VISIBLE_DEVICES="6,5,4,3,2,1,0"#UseUUIDstringsin`CUDA_VISIBLE_DEVICES`environmentvariable$exportCUDA_VISIBLE_DEVICES="$(nvisel-Ouuid-c2-f5000M)"CUDA_VISIBLE_DEVICES="GPU-849d5a8d-610e-eeea-1fd4-81ff44a23794,GPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1,GPU-96de99c9-d68f-84c8-424c-7c75e59cc0a0,GPU-2428d171-8684-5b64-830c-435cd972ec4a,GPU-6d2a57c9-7783-44bb-9f53-13f36282830a,GPU-f8e5a624-2c7e-417c-e647-b764d26d4733,GPU-f9ca790e-683e-3d56-00ba-8f654e977e02"#Pipeoutputtoothershellutilities$nvisel--newline-Ouuid-C6-f8GiBGPU-849d5a8d-610e-eeea-1fd4-81ff44a23794GPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1GPU-96de99c9-d68f-84c8-424c-7c75e59cc0a0GPU-2428d171-8684-5b64-830c-435cd972ec4aGPU-6d2a57c9-7783-44bb-9f53-13f36282830aGPU-f8e5a624-2c7e-417c-e647-b764d26d4733$nvisel-0-Ouuid-c2-f4GiB|xargs-0-I{}nvidia-smi--id={}--query-gpu=index,memory.free--format=csvCUDA_VISIBLE_DEVICES="GPU-849d5a8d-610e-eeea-1fd4-81ff44a23794,GPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1,GPU-96de99c9-d68f-84c8-424c-7c75e59cc0a0,GPU-2428d171-8684-5b64-830c-435cd972ec4a,GPU-6d2a57c9-7783-44bb-9f53-13f36282830a,GPU-f8e5a624-2c7e-417c-e647-b764d26d4733,GPU-f9ca790e-683e-3d56-00ba-8f654e977e02"index, memory.free [MiB]6, 11018 MiBindex, memory.free [MiB]5, 11018 MiBindex, memory.free [MiB]4, 11018 MiBindex, memory.free [MiB]3, 11018 MiBindex, memory.free [MiB]2, 11018 MiBindex, memory.free [MiB]1, 11018 MiBindex, memory.free [MiB]0, 11018 MiB#Normalizethe`CUDA_VISIBLE_DEVICES`environmentvariable(e.g.convertUUIDstoindicesorgetfullUUIDsforanabbreviatedform)$nvisel-i"GPU-18ef14e9,GPU-849d5a8d"-S5,6$nvisel-i"GPU-18ef14e9,GPU-849d5a8d"-S-Ouuid--newlineGPU-18ef14e9-dec6-1d7e-1284-3010c6ce98b1GPU-849d5a8d-610e-eeea-1fd4-81ff44a23794You can also integratenviselinto your training script like this:# Put this at the top of the Python scriptimportosfromnvitopimportselect_devicesos.environ['CUDA_VISIBLE_DEVICES']=','.join(select_devices(format='uuid',min_count=4,min_free_memory='8GiB'))Typenvisel --helpfor more command options:usage: nvisel [--help] [--version]
              [--inherit [CUDA_VISIBLE_DEVICES]] [--account-as-free [USERNAME ...]]
              [--min-count N] [--max-count N] [--count N]
              [--min-free-memory SIZE] [--min-total-memory SIZE]
              [--max-gpu-utilization RATE] [--max-memory-utilization RATE]
              [--tolerance TOL]
              [--format FORMAT] [--sep SEP | --newline | --null] [--no-sort]

CUDA visible devices selection tool.

options:
  --help, -h            Show this help message and exit.
  --version, -V         Show nvisel's version number and exit.

constraints:
  --inherit [CUDA_VISIBLE_DEVICES], -i [CUDA_VISIBLE_DEVICES]
                        Inherit the given `CUDA_VISIBLE_DEVICES`. If the argument is omitted, use the
                        value from the environment. This means selecting a subset of the currently
                        CUDA-visible devices.
  --account-as-free [USERNAME ...]
                        Account the used GPU memory of the given users as free memory.
                        If this option is specified but without argument, `$USER` will be used.
  --min-count N, -c N   Minimum number of devices to select. (default: 0)
                        The tool will fail (exit non-zero) if the requested resource is not available.
  --max-count N, -C N   Maximum number of devices to select. (default: all devices)
  --count N, -n N       Overriding both `--min-count N` and `--max-count N`.
  --min-free-memory SIZE, -f SIZE
                        Minimum free memory of devices to select. (example value: 4GiB)
                        If this constraint is given, check against all devices.
  --min-total-memory SIZE, -t SIZE
                        Minimum total memory of devices to select. (example value: 10GiB)
                        If this constraint is given, check against all devices.
  --max-gpu-utilization RATE, -G RATE
                        Maximum GPU utilization rate of devices to select. (example value: 30)
                        If this constraint is given, check against all devices.
  --max-memory-utilization RATE, -M RATE
                        Maximum memory bandwidth utilization rate of devices to select. (example value: 50)
                        If this constraint is given, check against all devices.
  --tolerance TOL, --tol TOL
                        The constraints tolerance (in percentage). (default: 0, i.e., strict)
                        This option can loose the constraints if the requested resource is not available.
                        For example, set `--tolerance=20` will accept a device with only 4GiB of free
                        memory when set `--min-free-memory=5GiB`.

formatting:
  --format FORMAT, -O FORMAT
                        The output format of the selected device identifiers. (default: index)
                        If any MIG device found, the output format will be fallback to `uuid`.
  --sep SEP, --separator SEP, -s SEP
                        Separator for the output. (default: ',')
  --newline             Use newline character as separator for the output, equivalent to `--sep=$'\n'`.
  --null, -0            Use null character ('\x00') as separator for the output. This option corresponds
                        to the `-0` option of `xargs`.
  --no-sort, -S         Do not sort the device by memory usage and GPU utilization.Callback Functions for Machine Learning Frameworksnvitopprovides two builtin callbacks forTensorFlow (Keras)andPyTorch Lightning.Callback forTensorFlow (Keras)fromtensorflow.python.keras.utils.multi_gpu_utilsimportmulti_gpu_modelfromtensorflow.python.keras.callbacksimportTensorBoardfromnvitop.callbacks.kerasimportGpuStatsLoggergpus=['/gpu:0','/gpu:1']# or `gpus = [0, 1]` or `gpus = 2`model=Xception(weights=None,..)model=multi_gpu_model(model,gpus)# optionalmodel.compile(..)tb_callback=TensorBoard(log_dir='./logs')# or `keras.callbacks.CSVLogger`gpu_stats=GpuStatsLogger(gpus)model.fit(..,callbacks=[gpu_stats,tb_callback])NOTE:Users should assign akeras.callbacks.TensorBoardcallback or akeras.callbacks.CSVLoggercallback to the model. And theGpuStatsLoggercallback should be placed before thekeras.callbacks.TensorBoard/keras.callbacks.CSVLoggercallback.Callback forPyTorch Lightningfromlightning.pytorchimportTrainerfromnvitop.callbacks.lightningimportGpuStatsLoggergpu_stats=GpuStatsLogger()trainer=Trainer(gpus=[..],logger=True,callbacks=[gpu_stats])NOTE:Users should assign a logger to the trainer.TensorBoardIntegrationPlease refer toResource Metric Collectorfor an example.More than a Monitornvitopcan be easily integrated into other applications. You can usenvitopto make your own monitoring tools. The full API references host athttps://nvitop.readthedocs.io.Quick StartA minimal script to monitor the GPU devices based on APIs fromnvitop:fromnvitopimportDevicedevices=Device.all()# or `Device.cuda.all()` to use CUDA ordinal insteadfordeviceindevices:processes=device.processes()# type: Dict[int, GpuProcess]sorted_pids=sorted(processes.keys())print(device)print(f'  - Fan speed:{device.fan_speed()}%')print(f'  - Temperature:{device.temperature()}C')print(f'  - GPU utilization:{device.gpu_utilization()}%')print(f'  - Total memory:{device.memory_total_human()}')print(f'  - Used memory:{device.memory_used_human()}')print(f'  - Free memory:{device.memory_free_human()}')print(f'  - Processes ({len(processes)}):{sorted_pids}')forpidinsorted_pids:print(f'    -{processes[pid]}')print('-'*120)Another more advanced approach with coloring:importtimefromnvitopimportDevice,GpuProcess,NA,coloredprint(colored(time.strftime('%a%b%d%H:%M:%S %Y'),color='red',attrs=('bold',)))devices=Device.cuda.all()# or `Device.all()` to use NVML ordinal insteadseparator=Falsefordeviceindevices:processes=device.processes()# type: Dict[int, GpuProcess]print(colored(str(device),color='green',attrs=('bold',)))print(colored('  - Fan speed:       ',color='blue',attrs=('bold',))+f'{device.fan_speed()}%')print(colored('  - Temperature:     ',color='blue',attrs=('bold',))+f'{device.temperature()}C')print(colored('  - GPU utilization: ',color='blue',attrs=('bold',))+f'{device.gpu_utilization()}%')print(colored('  - Total memory:    ',color='blue',attrs=('bold',))+f'{device.memory_total_human()}')print(colored('  - Used memory:     ',color='blue',attrs=('bold',))+f'{device.memory_used_human()}')print(colored('  - Free memory:     ',color='blue',attrs=('bold',))+f'{device.memory_free_human()}')iflen(processes)>0:processes=GpuProcess.take_snapshots(processes.values(),failsafe=True)processes.sort(key=lambdaprocess:(process.username,process.pid))print(colored(f'  - Processes ({len(processes)}):',color='blue',attrs=('bold',)))fmt='{pid:<5}{username:<8}{cpu:>5}{host_memory:>8}{time:>8}{gpu_memory:>8}{sm:>3}{command:<}'.formatprint(colored(fmt(pid='PID',username='USERNAME',cpu='CPU%',host_memory='HOST-MEM',time='TIME',gpu_memory='GPU-MEM',sm='SM%',command='COMMAND'),attrs=('bold',)))forsnapshotinprocesses:print(fmt(pid=snapshot.pid,username=snapshot.username[:7]+('+'iflen(snapshot.username)>8elsesnapshot.username[7:8]),cpu=snapshot.cpu_percent,host_memory=snapshot.host_memory_human,time=snapshot.running_time_human,gpu_memory=(snapshot.gpu_memory_humanifsnapshot.gpu_memory_humanisnotNAelse'WDDM:N/A'),sm=snapshot.gpu_sm_utilization,command=snapshot.command))else:print(colored('  - No Running Processes',attrs=('bold',)))ifseparator:print('-'*120)separator=TrueAn example monitoring script built with APIs fromnvitop.Status Snapshotnvitopprovides a helper functiontake_snapshotsto retrieve the status of both GPU devices and GPU processes at once. You can typehelp(nvitop.take_snapshots)in Python REPL for detailed documentation.In[1]:fromnvitopimporttake_snapshots,Device...:importos...:os.environ['CUDA_DEVICE_ORDER']='PCI_BUS_ID'...:os.environ['CUDA_VISIBLE_DEVICES']='1,0'# comma-separated integers or UUID stringsIn[2]:take_snapshots()# equivalent to `take_snapshots(Device.all())`Out[2]:SnapshotResult(devices=[DeviceSnapshot(real=Device(index=0,...),...),...],gpu_processes=[GpuProcessSnapshot(real=GpuProcess(pid=xxxxxx,device=Device(index=0,...),...),...),...])In[3]:device_snapshots,gpu_process_snapshots=take_snapshots(Device.all())# type: Tuple[List[DeviceSnapshot], List[GpuProcessSnapshot]]In[4]:device_snapshots,_=take_snapshots(gpu_processes=False)# ignore process snapshotsIn[5]:take_snapshots(Device.cuda.all())# use CUDA device enumerationOut[5]:SnapshotResult(devices=[CudaDeviceSnapshot(real=CudaDevice(cuda_index=0,nvml_index=1,...),...),CudaDeviceSnapshot(real=CudaDevice(cuda_index=1,nvml_index=0,...),...),],gpu_processes=[GpuProcessSnapshot(real=GpuProcess(pid=xxxxxx,device=CudaDevice(cuda_index=0,...),...),...),...])In[6]:take_snapshots(Device.cuda(1))# <CUDA 1> onlyOut[6]:SnapshotResult(devices=[CudaDeviceSnapshot(real=CudaDevice(cuda_index=1,nvml_index=0,...),...)],gpu_processes=[GpuProcessSnapshot(real=GpuProcess(pid=xxxxxx,device=CudaDevice(cuda_index=1,...),...),...),...])Please refer to sectionLow-level APIsfor more information.Resource Metric CollectorResourceMetricCollectoris a class that collects resource metrics for host, GPUs and processes running on the GPUs. All metrics will be collected in an asynchronous manner. You can typehelp(nvitop.ResourceMetricCollector)in Python REPL for detailed documentation.In[1]:fromnvitopimportResourceMetricCollector,Device...:importos...:os.environ['CUDA_DEVICE_ORDER']='PCI_BUS_ID'...:os.environ['CUDA_VISIBLE_DEVICES']='3,2,1,0'# comma-separated integers or UUID stringsIn[2]:collector=ResourceMetricCollector()# log all devices and descendant processes of the current process on the GPUsIn[3]:collector=ResourceMetricCollector(root_pids={1})# log all devices and all GPU processesIn[4]:collector=ResourceMetricCollector(devices=Device(0),root_pids={1})# log <GPU 0> and all GPU processes on <GPU 0>In[5]:collector=ResourceMetricCollector(devices=Device.cuda.all())# use the CUDA ordinalIn[6]:withcollector(tag='<tag>'):...:# Do something...:collector.collect()# -> Dict[str, float]# key -> '<tag>/<scope>/<metric (unit)>/<mean/min/max>'{'<tag>/host/cpu_percent (%)/mean':8.967849777683456,'<tag>/host/cpu_percent (%)/min':6.1,'<tag>/host/cpu_percent (%)/max':28.1,...,'<tag>/host/memory_percent (%)/mean':21.5,'<tag>/host/swap_percent (%)/mean':0.3,'<tag>/host/memory_used (GiB)/mean':91.0136418208109,'<tag>/host/load_average (%) (1 min)/mean':10.251427386878328,'<tag>/host/load_average (%) (5 min)/mean':10.072539414569503,'<tag>/host/load_average (%) (15 min)/mean':11.91126970422139,...,'<tag>/cuda:0 (gpu:3)/memory_used (MiB)/mean':3.875,'<tag>/cuda:0 (gpu:3)/memory_free (MiB)/mean':11015.562499999998,'<tag>/cuda:0 (gpu:3)/memory_total (MiB)/mean':11019.437500000002,'<tag>/cuda:0 (gpu:3)/memory_percent (%)/mean':0.0,'<tag>/cuda:0 (gpu:3)/gpu_utilization (%)/mean':0.0,'<tag>/cuda:0 (gpu:3)/memory_utilization (%)/mean':0.0,'<tag>/cuda:0 (gpu:3)/fan_speed (%)/mean':22.0,'<tag>/cuda:0 (gpu:3)/temperature (C)/mean':25.0,'<tag>/cuda:0 (gpu:3)/power_usage (W)/mean':19.11166264116916,...,'<tag>/cuda:1 (gpu:2)/memory_used (MiB)/mean':8878.875,...,'<tag>/cuda:2 (gpu:1)/memory_used (MiB)/mean':8182.875,...,'<tag>/cuda:3 (gpu:0)/memory_used (MiB)/mean':9286.875,...,'<tag>/pid:12345/host/cpu_percent (%)/mean':151.34342772112265,'<tag>/pid:12345/host/host_memory (MiB)/mean':44749.72373447514,'<tag>/pid:12345/host/host_memory_percent (%)/mean':8.675082352111717,'<tag>/pid:12345/host/running_time (min)':336.23803206741576,'<tag>/pid:12345/cuda:1 (gpu:4)/gpu_memory (MiB)/mean':8861.0,'<tag>/pid:12345/cuda:1 (gpu:4)/gpu_memory_percent (%)/mean':80.4,'<tag>/pid:12345/cuda:1 (gpu:4)/gpu_memory_utilization (%)/mean':6.711118172407917,'<tag>/pid:12345/cuda:1 (gpu:4)/gpu_sm_utilization (%)/mean':48.23283397736476,...,'<tag>/duration (s)':7.247399162035435,'<tag>/timestamp':1655909466.9981883}The results can be easily logged intoTensorBoardor a CSV file. For example:importosimporttorchimporttorch.nnasnnimporttorch.nn.functionalasFfromtorch.utils.tensorboardimportSummaryWriterfromnvitopimportCudaDevice,ResourceMetricCollectorfromnvitop.callbacks.tensorboardimportadd_scalar_dict# Build networks and prepare datasets...# Logger and status collectorwriter=SummaryWriter()collector=ResourceMetricCollector(devices=CudaDevice.all(),# log all visible CUDA devices and use the CUDA ordinalroot_pids={os.getpid()},# only log the descendant processes of the current processinterval=1.0)# snapshot interval for background daemon thread# Start trainingglobal_step=0forepochinrange(num_epoch):withcollector(tag='train'):forbatchintrain_dataset:withcollector(tag='batch'):metrics=train(net,batch)global_step+=1add_scalar_dict(writer,'train',metrics,global_step=global_step)add_scalar_dict(writer,'resources',# tag='resources/train/batch/...'collector.collect(),global_step=global_step)add_scalar_dict(writer,'resources',# tag='resources/train/...'collector.collect(),global_step=epoch)withcollector(tag='validate'):metrics=validate(net,validation_dataset)add_scalar_dict(writer,'validate',metrics,global_step=epoch)add_scalar_dict(writer,'resources',# tag='resources/validate/...'collector.collect(),global_step=epoch)Another example for logging into a CSV file:importdatetimeimporttimeimportpandasaspdfromnvitopimportResourceMetricCollectorcollector=ResourceMetricCollector(root_pids={1},interval=2.0)# log all devices and all GPU processesdf=pd.DataFrame()withcollector(tag='resources'):for_inrange(60):# Do somethingtime.sleep(60)metrics=collector.collect()df_metrics=pd.DataFrame.from_records(metrics,index=[len(df)])df=pd.concat([df,df_metrics],ignore_index=True)# Flush to CSV file ...df.insert(0,'time',df['resources/timestamp'].map(datetime.datetime.fromtimestamp))df.to_csv('results.csv',index=False)You can also daemonize the collector in the background usingcollect_in_backgroundorResourceMetricCollector.daemonizewith callback functions.fromnvitopimportDevice,ResourceMetricCollector,collect_in_backgroundlogger=...defon_collect(metrics):# will be called periodicallyiflogger.is_closed():# closed manually by userreturnFalselogger.log(metrics)returnTruedefon_stop(collector):# will be called only once at stopifnotlogger.is_closed():logger.close()# cleanup# Record metrics to the logger in the background every 5 seconds.# It will collect 5-second mean/min/max for each metric.collect_in_background(on_collect,ResourceMetricCollector(Device.cuda.all()),interval=5.0,on_stop=on_stop,)or simply:ResourceMetricCollector(Device.cuda.all()).daemonize(on_collect,interval=5.0,on_stop=on_stop,)Low-level APIsThe full API references can be found athttps://nvitop.readthedocs.io.DeviceThedevice moduleprovides:Device([index, uuid, bus_id])Live class of the GPU devices, different from the device snapshots.PhysicalDevice([index, uuid, bus_id])Class for physical devices.MigDevice([index, uuid, bus_id])Class for MIG devices.CudaDevice([cuda_index, nvml_index, uuid])Class for devices enumerated over the CUDA ordinal.CudaMigDevice([cuda_index, nvml_index, uuid])Class for CUDA devices that are MIG devices.parse_cuda_visible_devices([...])Parse the givenCUDA_VISIBLE_DEVICESvalue into a list of NVML device indices.normalize_cuda_visible_devices([...])Parse the givenCUDA_VISIBLE_DEVICESvalue and convert it into a comma-separated string of UUIDs.In[1]:fromnvitopimport(...:host,...:Device,PhysicalDevice,CudaDevice,...:parse_cuda_visible_devices,normalize_cuda_visible_devices...:HostProcess,GpuProcess,...:NA,...:)...:importos...:os.environ['CUDA_DEVICE_ORDER']='PCI_BUS_ID'...:os.environ['CUDA_VISIBLE_DEVICES']='9,8,7,6'# comma-separated integers or UUID stringsIn[2]:Device.driver_version()Out[2]:'525.60.11'In[3]:Device.cuda_driver_version()# the maximum CUDA version supported by the driver (can be different from the CUDA Runtime version)Out[3]:'12.0'In[4]:Device.cuda_runtime_version()# the CUDA Runtime versionOut[4]:'11.8'In[5]:Device.count()Out[5]:10In[6]:CudaDevice.count()# or `Device.cuda.count()`Out[6]:4In[7]:all_devices=Device.all()# all devices on board (physical device)...:nvidia0,nvidia1=Device.from_indices([0,1])# from physical device indices...:all_devicesOut[7]:[PhysicalDevice(index=0,name="GeForce RTX 2080 Ti",total_memory=11019MiB),PhysicalDevice(index=1,name="GeForce RTX 2080 Ti",total_memory=11019MiB),PhysicalDevice(index=2,name="GeForce RTX 2080 Ti",total_memory=11019MiB),PhysicalDevice(index=3,name="GeForce RTX 2080 Ti",total_memory=11019MiB),PhysicalDevice(index=4,name="GeForce RTX 2080 Ti",total_memory=11019MiB),PhysicalDevice(index=5,name="GeForce RTX 2080 Ti",total_memory=11019MiB),PhysicalDevice(index=6,name="GeForce RTX 2080 Ti",total_memory=11019MiB),PhysicalDevice(index=7,name="GeForce RTX 2080 Ti",total_memory=11019MiB),PhysicalDevice(index=8,name="GeForce RTX 2080 Ti",total_memory=11019MiB),PhysicalDevice(index=9,name="GeForce RTX 2080 Ti",total_memory=11019MiB)]In[8]:# NOTE: The function results might be different between calls when the `CUDA_VISIBLE_DEVICES` environment variable has been modified...:cuda_visible_devices=Device.from_cuda_visible_devices()# from the `CUDA_VISIBLE_DEVICES` environment variable...:cuda0,cuda1=Device.from_cuda_indices([0,1])# from CUDA device indices (might be different from physical device indices if `CUDA_VISIBLE_DEVICES` is set)...:cuda_visible_devices=CudaDevice.all()# shortcut to `Device.from_cuda_visible_devices()`...:cuda_visible_devices=Device.cuda.all()# `Device.cuda` is aliased to `CudaDevice`...:cuda_visible_devicesOut[8]:[CudaDevice(cuda_index=0,nvml_index=9,name="NVIDIA GeForce RTX 2080 Ti",total_memory=11019MiB),CudaDevice(cuda_index=1,nvml_index=8,name="NVIDIA GeForce RTX 2080 Ti",total_memory=11019MiB),CudaDevice(cuda_index=2,nvml_index=7,name="NVIDIA GeForce RTX 2080 Ti",total_memory=11019MiB),CudaDevice(cuda_index=3,nvml_index=6,name="NVIDIA GeForce RTX 2080 Ti",total_memory=11019MiB)]In[9]:nvidia0=Device(0)# from device index (or `Device(index=0)`)...:nvidia0Out[9]:PhysicalDevice(index=0,name="GeForce RTX 2080 Ti",total_memory=11019MiB)In[10]:nvidia1=Device(uuid='GPU-01234567-89ab-cdef-0123-456789abcdef')# from UUID string (or just `Device('GPU-xxxxxxxx-...')`)...:nvidia2=Device(bus_id='00000000:06:00.0')# from PCI bus ID...:nvidia1Out[10]:PhysicalDevice(index=1,name="GeForce RTX 2080 Ti",total_memory=11019MiB)In[11]:cuda0=CudaDevice(0)# from CUDA device index (equivalent to `CudaDevice(cuda_index=0)`)...:cuda1=CudaDevice(nvml_index=8)# from physical device index...:cuda3=CudaDevice(uuid='GPU-xxxxxxxx-...')# from UUID string...:cuda4=Device.cuda(4)# `Device.cuda` is aliased to `CudaDevice`...:cuda0Out[11]:CudaDevice(cuda_index=0,nvml_index=9,name="NVIDIA GeForce RTX 2080 Ti",total_memory=11019MiB)In[12]:nvidia0.memory_used()# in bytesOut[12]:9293398016In[13]:nvidia0.memory_used_human()Out[13]:'8862MiB'In[14]:nvidia0.gpu_utilization()# in percentageOut[14]:5In[15]:nvidia0.processes()# type: Dict[int, GpuProcess]Out[15]:{52059:GpuProcess(pid=52059,gpu_memory=7885MiB,type=C,device=PhysicalDevice(index=0,name="GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=52059,name='ipython3',status='sleeping',started='14:31:22')),53002:GpuProcess(pid=53002,gpu_memory=967MiB,type=C,device=PhysicalDevice(index=0,name="GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=53002,name='python',status='running',started='14:31:59'))}In[16]:nvidia1_snapshot=nvidia1.as_snapshot()...:nvidia1_snapshotOut[16]:PhysicalDeviceSnapshot(real=PhysicalDevice(index=1,name="GeForce RTX 2080 Ti",total_memory=11019MiB),bus_id='00000000:05:00.0',compute_mode='Default',clock_infos=ClockInfos(graphics=1815,sm=1815,memory=6800,video=1680),# in MHzclock_speed_infos=ClockSpeedInfos(current=ClockInfos(graphics=1815,sm=1815,memory=6800,video=1680),max=ClockInfos(graphics=2100,sm=2100,memory=7000,video=1950)),# in MHzcuda_compute_capability=(7,5),current_driver_model='N/A',decoder_utilization=0,# in percentagedisplay_active='Disabled',display_mode='Disabled',encoder_utilization=0,# in percentagefan_speed=22,# in percentagegpu_utilization=17,# in percentage (NOTE: this is the utilization rate of SMs, i.e. GPU percent)index=1,max_clock_infos=ClockInfos(graphics=2100,sm=2100,memory=7000,video=1950),# in MHzmemory_clock=6800,# in MHzmemory_free=10462232576,# in bytesmemory_free_human='9977MiB',memory_info=MemoryInfo(total=11554717696,free=10462232576,used=1092485120)# in bytesmemory_percent=9.5,# in percentage (NOTE: this is the percentage of used GPU memory)memory_total=11554717696,# in bytesmemory_total_human='11019MiB',memory_usage='1041MiB / 11019MiB',memory_used=1092485120,# in bytesmemory_used_human='1041MiB',memory_utilization=7,# in percentage (NOTE: this is the utilization rate of GPU memory bandwidth)mig_mode='N/A',name='GeForce RTX 2080 Ti',pcie_rx_throughput=1000,# in KiB/spcie_rx_throughput_human='1000KiB/s',pcie_throughput=ThroughputInfo(tx=1000,rx=1000),# in KiB/spcie_tx_throughput=1000,# in KiB/spcie_tx_throughput_human='1000KiB/s',performance_state='P2',persistence_mode='Disabled',power_limit=250000,# in milliwatts (mW)power_status='66W / 250W',# in watts (W)power_usage=66051,# in milliwatts (mW)sm_clock=1815,# in MHztemperature=39,# in Celsiustotal_volatile_uncorrected_ecc_errors='N/A',utilization_rates=UtilizationRates(gpu=17,memory=7,encoder=0,decoder=0),# in percentageuuid='GPU-01234567-89ab-cdef-0123-456789abcdef',)In[17]:nvidia1_snapshot.memory_percent# snapshot uses properties instead of function callsOut[17]:9.5In[18]:nvidia1_snapshot['memory_info']# snapshot also supports `__getitem__` by stringOut[18]:MemoryInfo(total=11554717696,free=10462232576,used=1092485120)In[19]:nvidia1_snapshot.bar1_memory_info# snapshot will automatically retrieve not presented attributes from `real`Out[19]:MemoryInfo(total=268435456,free=257622016,used=10813440)NOTE:Some entry values may be'N/A'(type:NaType, a subclass ofstr) when the corresponding resources are not applicable. TheNAvalue supports arithmetic operations. It acts likemath.nan: float.>>>fromnvitopimportNA>>>NA'N/A'>>>'memory usage:{}'.format(NA)# NA is an instance of `str`'memory usage: N/A'>>>NA.lower()# NA is an instance of `str`'n/a'>>>NA.ljust(5)# NA is an instance of `str`'N/A  '>>>NA+'str'# string contamination if the operand is a string'N/Astr'>>>float(NA)# explicit conversion to float (`math.nan`)nan>>>NA+1# auto-casting to float if the operand is a numbernan>>>NA*1024# auto-casting to float if the operand is a numbernan>>>NA/(1024*1024)# auto-casting to float if the operand is a numbernanYou can useentry != 'N/A'conditions to avoid exceptions. It's safe to usefloat(entry)for numbers whileNaTypewill be converted tomath.nan. For example:memory_used:Union[int,NaType]=device.memory_used()# memory usage in bytes or `'N/A'`memory_used_in_mib:float=float(memory_used)/(1<<20)# memory usage in Mebibytes (MiB) or `math.nan`It's safe to compareNaTypewith numbers, butNaTypeis always larger than any number:devices_by_used_memory=sorted(Device.all(),key=Device.memory_used,reverse=True)# it's safe to compare `'N/A'` with numbersdevices_by_free_memory=sorted(Device.all(),key=Device.memory_free,reverse=True)# please add `memory_free != 'N/A'` checks if sort in descending order hereSeenvitop.NaTypedocumentation for more details.ProcessTheprocess moduleprovides:HostProcess([pid])Represents an OS process with the given PID.GpuProcess(pid, device[, gpu_memory, ...])Represents a process with the given PID running on the given GPU device.command_join(cmdline)Returns a shell-escaped string from command line arguments.In[20]:processes=nvidia1.processes()# type: Dict[int, GpuProcess]...:processesOut[20]:{23266:GpuProcess(pid=23266,gpu_memory=1031MiB,type=C,device=Device(index=1,name="GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=23266,name='python3',status='running',started='2021-05-10 21:02:40'))}In[21]:process=processes[23266]...:processOut[21]:GpuProcess(pid=23266,gpu_memory=1031MiB,type=C,device=Device(index=1,name="GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=23266,name='python3',status='running',started='2021-05-10 21:02:40'))In[22]:process.status()# GpuProcess will automatically inherit attributes from GpuProcess.hostOut[22]:'running'In[23]:process.cmdline()# type: List[str]Out[23]:['python3','rllib_train.py']In[24]:process.command()# type: strOut[24]:'python3 rllib_train.py'In[25]:process.cwd()# GpuProcess will automatically inherit attributes from GpuProcess.hostOut[25]:'/home/xxxxxx/Projects/xxxxxx'In[26]:process.gpu_memory_human()Out[26]:'1031MiB'In[27]:process.as_snapshot()Out[27]:GpuProcessSnapshot(real=GpuProcess(pid=23266,gpu_memory=1031MiB,type=C,device=PhysicalDevice(index=1,name="GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=23266,name='python3',status='running',started='2021-05-10 21:02:40')),cmdline=['python3','rllib_train.py'],command='python3 rllib_train.py',compute_instance_id='N/A',cpu_percent=98.5,# in percentagedevice=PhysicalDevice(index=1,name="GeForce RTX 2080 Ti",total_memory=11019MiB),gpu_encoder_utilization=0,# in percentagegpu_decoder_utilization=0,# in percentagegpu_instance_id='N/A',gpu_memory=1081081856,# in bytesgpu_memory_human='1031MiB',gpu_memory_percent=9.4,# in percentage (NOTE: this is the percentage of used GPU memory)gpu_memory_utilization=5,# in percentage (NOTE: this is the utilization rate of GPU memory bandwidth)gpu_sm_utilization=0,# in percentage (NOTE: this is the utilization rate of SMs, i.e. GPU percent)host=HostProcessSnapshot(real=HostProcess(pid=23266,name='python3',status='running',started='2021-05-10 21:02:40'),cmdline=['python3','rllib_train.py'],command='python3 rllib_train.py',cpu_percent=98.5,# in percentagehost_memory=9113627439,# in byteshost_memory_human='8691MiB',is_running=True,memory_percent=1.6849018430285683,# in percentagename='python3',running_time=datetime.timedelta(days=1,seconds=80013,microseconds=470024),running_time_human='46:13:33',running_time_in_seconds=166413.470024,status='running',username='panxuehai',),host_memory=9113627439,# in byteshost_memory_human='8691MiB',is_running=True,memory_percent=1.6849018430285683,# in percentage (NOTE: this is the percentage of used host memory)name='python3',pid=23266,running_time=datetime.timedelta(days=1,seconds=80013,microseconds=470024),running_time_human='46:13:33',running_time_in_seconds=166413.470024,status='running',type='C',# 'C' for Compute / 'G' for Graphics / 'C+G' for Bothusername='panxuehai',)In[28]:process.uids()# GpuProcess will automatically inherit attributes from GpuProcess.hostOut[28]:puids(real=1001,effective=1001,saved=1001)In[29]:process.kill()# GpuProcess will automatically inherit attributes from GpuProcess.hostIn[30]:list(map(Device.processes,all_devices))# all processesOut[30]:[{52059:GpuProcess(pid=52059,gpu_memory=7885MiB,type=C,device=PhysicalDevice(index=0,name="GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=52059,name='ipython3',status='sleeping',started='14:31:22')),53002:GpuProcess(pid=53002,gpu_memory=967MiB,type=C,device=PhysicalDevice(index=0,name="GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=53002,name='python',status='running',started='14:31:59'))},{},{},{},{},{},{},{},{84748:GpuProcess(pid=84748,gpu_memory=8975MiB,type=C,device=PhysicalDevice(index=8,name="GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=84748,name='python',status='running',started='11:13:38'))},{84748:GpuProcess(pid=84748,gpu_memory=8341MiB,type=C,device=PhysicalDevice(index=9,name="GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=84748,name='python',status='running',started='11:13:38'))}]In[31]:this=HostProcess(os.getpid())...:thisOut[31]:HostProcess(pid=35783,name='python',status='running',started='19:19:00')In[32]:this.cmdline()# type: List[str]Out[32]:['python','-c','import IPython; IPython.terminal.ipapp.launch_new_instance()']In[33]:this.command()# not simply `' '.join(cmdline)` but quotes are addedOut[33]:'python -c "import IPython; IPython.terminal.ipapp.launch_new_instance()"'In[34]:this.memory_info()Out[34]:pmem(rss=83988480,vms=343543808,shared=12079104,text=8192,lib=0,data=297435136,dirty=0)In[35]:importcupyascp...:x=cp.zeros((10000,1000))...:this=GpuProcess(os.getpid(),cuda0)# construct from `GpuProcess(pid, device)` explicitly rather than calling `device.processes()`...:thisOut[35]:GpuProcess(pid=35783,gpu_memory=N/A,type=N/A,device=CudaDevice(cuda_index=0,nvml_index=9,name="NVIDIA GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=35783,name='python',status='running',started='19:19:00'))In[36]:this.update_gpu_status()# update used GPU memory from new driver queriesOut[36]:267386880In[37]:thisOut[37]:GpuProcess(pid=35783,gpu_memory=255MiB,type=C,device=CudaDevice(cuda_index=0,nvml_index=9,name="NVIDIA GeForce RTX 2080 Ti",total_memory=11019MiB),host=HostProcess(pid=35783,name='python',status='running',started='19:19:00'))In[38]:id(this)==id(GpuProcess(os.getpid(),cuda0))# IMPORTANT: the instance will be reused while the process is runningOut[38]:TrueHost (inherited frompsutil)In[39]:host.cpu_count()Out[39]:88In[40]:host.cpu_percent()Out[40]:18.5In[41]:host.cpu_times()Out[41]:scputimes(user=2346377.62,nice=53321.44,system=579177.52,idle=10323719.85,iowait=28750.22,irq=0.0,softirq=11566.87,steal=0.0,guest=0.0,guest_nice=0.0)In[42]:host.load_average()Out[42]:(14.88,17.8,19.91)In[43]:host.virtual_memory()Out[43]:svmem(total=270352478208,available=192275968000,percent=28.9,used=53350518784,free=88924037120,active=125081112576,inactive=44803993600,buffers=37006450688,cached=91071471616,shared=23820632064,slab=8200687616)In[44]:host.memory_percent()Out[44]:28.9In[45]:host.swap_memory()Out[45]:sswap(total=65534947328,used=475136,free=65534472192,percent=0.0,sin=2404139008,sout=4259434496)In[46]:host.swap_percent()Out[46]:0.0ScreenshotsExample output ofnvitop -1:Example output ofnvitop:FullCompactTree-view screen (shortcut:t) for GPU processes and their ancestors:NOTE:The process tree is built in backward order (recursively back to the tree root). Only GPU processes along with their children and ancestors (parents and grandparents ...) will be shown. Not all running processes will be displayed.Environment variable screen (shortcut:e):Spectrum-like bar charts (with option--colorful):ChangelogSeeCHANGELOG.md.LicenseThe source code ofnvitopis dual-licensed by theApache License, Version 2.0 (Apache-2.0)andGNU General Public License, Version 3 (GPL-3.0). ThenvitopCLI is released under theGPL-3.0license while the remaining part ofnvitopis released under theApache-2.0license. The license files can be found atLICENSE(Apache-2.0) andCOPYING(GPL-3.0).The source code is organized as:nvitop           (GPL-3.0)
├── __init__.py  (Apache-2.0)
├── version.py   (Apache-2.0)
├── api          (Apache-2.0)
│   ├── LICENSE  (Apache-2.0)
│   └── *        (Apache-2.0)
├── callbacks    (Apache-2.0)
│   ├── LICENSE  (Apache-2.0)
│   └── *        (Apache-2.0)
├── select.py    (Apache-2.0)
├── __main__.py  (GPL-3.0)
├── cli.py       (GPL-3.0)
└── gui          (GPL-3.0)
    ├── COPYING  (GPL-3.0)
    └── *        (GPL-3.0)Copyright NoticePlease feel free to usenvitopas a dependency for your own projects. The following Python import statements are permitted:importnvitopimportnvitopasaliasimportnvitop.apiasapiimportnvitop.deviceasdevicefromnvitopimport*fromnvitop.apiimport*fromnvitopimportDevice,ResourceMetricCollectorThe public APIs fromnvitopare released under theApache License, Version 2.0 (Apache-2.0). The original license files can be found atLICENSE,nvitop/api/LICENSE, andnvitop/callbacks/LICENSE.The CLI ofnvitopis released under theGNU General Public License, Version 3 (GPL-3.0). The original license files can be found atCOPYINGandnvitop/gui/COPYING. If you dynamically load the source code ofnvitop's CLI or GUI:fromnvitopimportclifromnvitopimportguiimportnvitop.cliimportnvitop.guiyour source code should also be released under the GPL-3.0 License.If you want to add or modify some features ofnvitop's CLI, or copy some source code ofnvitop's CLI into your own code, the source code should also be released under the GPL-3.0 License (asnvitopcontains some modified source code fromrangerunder the GPL-3.0 License).